EmailRelay and OAUTH - GMAIL

Summary Description

Nightly CS Gold reports (and other emails) stopped sending because E-MailRelay could not authenticate to Gmail SMTP, causing messages to accumulate and age out in the E-MailRelay spool directory. The remediation involved correcting how the SMTP auth file was referenced (moving the auth file argument from the start script into emailrelay.conf) and ensuring Gmail credentials were an app password formatted in a way E-MailRelay can use.

Prerequisites

• OS-level access to the CS Gold server(s) involved (example server referenced: 1cardtps).
• Ability to restart Windows services.
• Ability to view and edit E-MailRelay configuration files.
• Gmail admin/user access to the sending mailbox (example: cbord@xyz.edu) to create/rotate an App Password.
• Ability to run commands from an elevated command prompt.

Contents

1. Symptoms
2. Root Cause
3. Diagnostics
4. Resolution
5. Validation
6. Rollback

Detailed Description

Symptoms

• No nightly reports received after a specific date (reported as “since 4/28/2025”).
• E-MailRelay spool directory continues to build up and messages “age out.”
• SMTP authentication failures shown in logs/errors, including:
  • smtp error: authentication failed using mechanism [login] ... [5.7.8 Username and Password not accepted ... BadCredentials]
• Restarting E-MailRelay service does not restore mail flow.
• Test emails may be rejected and remain in spool.

Root Cause

• Gmail SMTP authentication failed due to a mismatch between what E-MailRelay expects (simple username/password) and how credentials/config were applied.
• Two specific contributing issues surfaced in the case:
  • The auth file format/quoting and/or password format caused E-MailRelay to fail to start or fail auth. (Quotes were specifically called out as problematic; passwords with spaces were suspected problematic.)
  • The E-MailRelay auth file argument/parameter worked reliably when moved from emailrelay-start.bat into emailrelay.conf (last change that allowed the service to start “right up”).

Diagnostics

1. Confirm the mail pipeline and where it’s failing

• CS Gold uses Blat to generate emails, then hands off to E-MailRelay; E-MailRelay uses its spool directory as a holding area when delivery fails.
• Check spool directory for growth and file timestamps (example path referenced: D:\Program Files\emailrelay\spool).
  • If you see large counts of files and old timestamps, that confirms backlog rather than new successful deliveries.

2. Confirm the failure mode is authentication

• Look for SMTP auth errors indicating “BadCredentials” / “Username and Password not accepted.”
• If a test email is attempted and becomes “parked in spool” due to authentication rejection, the issue is still at SMTP auth.

3. Validate Gmail credential type

• Confirm the sending mailbox has MFA/2-step enabled (required to create app passwords in many org configurations).
• Confirm an App Password was generated for the mailbox (Gmail “App Passwords”) and that the value used is the generated app password (commonly a 16-character value), not the normal account password.

4. Check for inconsistent sender identity between Blat and E-MailRelay

• Run (elevated):
  • blat -profile
• If Blat is configured to use a different account (example concern raised: onecard@smcm.edu vs cbord@xyz.edu), update Blat’s profile to match the intended sender.
  • Note: The transcript indicates the blat -profile output was truncated, so confirm the configured profile values directly on the server.

5. Confirm where E-MailRelay reads authentication settings

• Identify whether --client-auth (or equivalent) is being provided in:
  • emailrelay-start.bat
  • emailrelay.conf
• In this case, E-MailRelay only stabilized after moving the auth-file argument from the .bat into emailrelay.conf.

Resolution

Important: Take backups before changing anything.

1. Backup current E-MailRelay configuration

• Copy these files to a safe location (same folder is fine if access-controlled):
  • emailrelay.conf → emailrelay.conf.pre_change
  • emailrelay-start.bat → emailrelay-start.bat.pre_change
  • emailrelay.auth (or your auth file name) → emailrelay.auth.pre_change

2. Confirm Gmail App Password use

• Generate a fresh Gmail App Password for the sender mailbox (example mailbox: cbord@smcm.edu).
• Use a password format compatible with E-MailRelay:
  • Avoid spaces.
  • Avoid quotes around the password in the auth file unless your E-MailRelay documentation explicitly confirms quoting is supported in your version.

3. Ensure E-MailRelay is referencing the auth file in the supported location

• If E-MailRelay fails to start or fails auth when --client-auth is passed in emailrelay-start.bat, move that argument into emailrelay.conf (this was the last change that restored E-MailRelay startup in the case).
• Verify that emailrelay.conf contains the auth reference and any required TLS settings for Gmail (TLS requirement was mentioned in the EmailRelay guidance shared during the case).

4. Restart E-MailRelay service

• Restart the E-MailRelay Windows service on the relevant server(s).
• If the environment has E-MailRelay on both TPS and DB servers (the transcript indicates impact on both), apply the same correction consistently.

5. Handle stale spool artifacts

• Identify .bad files and confirm they correspond to older failed deliveries (compare timestamps).
• Clear only confirmed stale .bad/failed items to reduce noise while preserving evidence if needed for RCA.
  • If retaining evidence is required, zip/archive the spool contents first.

Validation

1. Confirm E-MailRelay stays running

• In Windows Services, verify the E-MailRelay service remains started and does not stop immediately.

2. Confirm successful delivery path with a targeted Blat test

Run the one-off email test described in the case (edit recipient and paths to match your environment):

• Open an elevated command prompt.
• Navigate to a known log directory containing Nightly.* files (example pattern given: D:\CBORD\DCS\GoldServer\log\<date_time_folder>).
• Create a small body file and send the attachments using Blat:
• echo youremailhere >1
• for %f in (Nightly.*) do blat 1 -t <your_email@domain> -s "Test" -attach %f

Expected result:

• New messages appear in Gmail “Sent” / SMTP logs (if available) and are received by the target inbox.
• E-MailRelay spool does not accumulate new items for this test, or the count decreases after send attempts.

3. Confirm CS Gold nightly reports can re-run and deliver

• From the Gold bin directory, re-run nightly reports (as referenced in the transcript):
  • RptNightly.exe *,-1

Expected result:

• Emails are delivered and do not remain queued in spool.

4. Confirm backlog behavior

• If spool still contains older files, confirm the timestamps are historical and not newly generated.
• Verify that new test sends either do not create new spool items or those items clear quickly.

Rollback

Use rollback if:

• E-MailRelay cannot stay running after changes, OR
• Authentication failures worsen / service becomes unstable.

1. Restore backups

• Restore:
  • emailrelay.conf.pre_change → emailrelay.conf
  • emailrelay-start.bat.pre_change → emailrelay-start.bat
  • emailrelay.auth.pre_change → emailrelay.auth

2. Restart service

• Restart the E-MailRelay service.

3. Preserve evidence

• Archive spool contents and relevant logs before clearing anything, especially if escalation to SMTP/Gmail admins is required.

Notes from this case useful for future incidents:

• Gmail SMTP auth issues can present as E-MailRelay spool growth + “BadCredentials” errors.
• Quotes in the auth file were explicitly noted as something to avoid in this case, and passwords with spaces were suspected to break parsing.
• The most impactful stabilization change reported was moving the auth-file argument from emailrelay-start.bat into emailrelay.conf.
• If mail still fails after E-MailRelay is running, validate Blat configuration (blat -profile) to ensure it is not using an unexpected sender identity (example concern: onecard@smcm.edu).

Consolidated Follow-Up Questions (only if you want a tighter, more exact KBA)

• What is the exact E-MailRelay version and the exact service name on Windows?
• What are the current contents of emailrelay.conf, emailrelay-start.bat, and the auth file (mask secrets)?
• Can you provide the full (non-truncated) output of blat -profile?