Fail TeamCity on not resolve host

Problem

You have setup a job in TeamCity that runs an http request using curl. The job shows green but you’re not seeing the data or expected outcome. In the logs it says “Could not resolve host”. That’s obviously a separate issue, but a disappointing issue is that despite this failure the job reports green as though it succeeded.

Solution

I had to add the following

	set -x
	set -e
	ping -c 4 %host.name%
	curl -H "Accept: application/json" -H \
	http://%host.name%/GetStatus \
	| jq . | tee status.txt
	res=$?
	if test "$res" != "0"; then
	echo "the curl command failed with: $res"
	fi

view raw curl.sh hosted with ❤ by GitHub

The key is that ping will set the exit status code to 2 rather than 0. This will fail the build in TeamCity and make it obvious that there is in fact an issue.

References

1. Curl Exit Status
2. StackOverflow: Meaning of $?

Mongo Namespace Gotcha

Problem

You’re trying to restore a Mongo DB backup, but you’re getting an odd error like the one below.

	mongorestore –host mongoose –ssl –username user –password password –authenticationDatabase myDB –archive=/dump_myDB.archive –gzip –nsInclude "myDB" –nsFrom "myDB" –nsTo "myDB"
	2019-08-28T11:15:40.909-0400 preparing collections to restore from
	2019-08-28T11:17:22.461-0400 done
	mongorestore –host mongoose –ssl –username user –password password –authenticationDatabase myDB –archive=/dump_myDB.archive –gzip –nsFrom "myDB" –nsTo "myDB"
	2019-08-28T11:43:53.381-0400 preparing collections to restore from
	2019-08-28T11:43:53.440-0400 Failed: myDB.1359_23_10591_1290: error reading database: not authorized on myDB to execute command { listCollections: 1, cursor: { batchSize: 0 }, $db: "myDB" }

view raw mongorestore.sh hosted with ❤ by GitHub

Solution

You need to add .* at the end of all namepsace parameters like the below example.

mongorestore --host mongoose --ssl --username user --password password --authenticationDatabase myDB --archive=/dump_myDB.archive --gzip --nsInclude "myDB.*” --nsFrom "myDB.*” --nsTo "myDB.*”

References

1. mongorestore –nsInclude=<namespace pattern>

Mongo SSL Gotcha

Problem

You’re trying to connect to a mongodb instance using the mongo shell. However you’re getting a cryptic error like the one below.

	..[ryan@local] – [~/Documents/git/repo] – [Tue Aug 27, 04:25]
	..[$] <((git)-[branch]-)> mongo –host=mongohost
	MongoDB shell version v4.0.3
	connecting to: mongodb://mongohost:27017/
	2019-08-27T16:25:35.608-0400 E QUERY [js] Error: network error while attempting to run command 'isMaster' on hmongohost:27017' : connect@src/mongo/shell/mongo.js:257:13
	@(connect):1:6
	exception: connect failed

view raw shell.sh hosted with ❤ by GitHub

Solution

The key here is to add the --ssl flag like the below

mongo --host=mongohost --ssl

References

1. Connecting to MongoDB fails
2. MongoDocs: –ssl

Bitbucket Markdown

Problem

I wanted to create a Readme.md file for future engineers to use to get up and running in Bitbucket. One of the steps was to install chocolatey.

@"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

but it showed up like

Solution

Had to add \\ instead of \ to the powershell.exe path

`@"%SystemRoot%\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" -NoProfile -InputFormat None -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"`

which corrected the preview to be like the below image

Sensu Check Hook Gotcha

Problem

Sensu check hooks are a great tool for auto-remediation. There may be cases were you want to attempt multiple actions or scripts to resolve the issue. Say you’re trying to run two scripts from sensu check hooks on sensu 1.7 on Windows. The below does not run both, but will only execute the first script.

powershell -file \"c:\opt\sensu\plugins\custom_script_1.ps1\" ; powershell -file \"c:\opt\sensu\plugins\custom_script_2.ps1\"

Solution

The below command runs both regardless of whether the first fails or is successful.

powershell -file \"c:\opt\sensu\plugins\custom_script_1.ps1\" & powershell -file \"c:\opt\sensu\plugins\custom_script_2.ps1\"

The below command will only run the second command if the first succeeds.

powershell -file \"c:\opt\sensu\plugins\custom_script_1.ps1\" && powershell -file \"c:\opt\sensu\plugins\custom_script_2.ps1\"

Use Data to Trim alerts

Problem

I leverage sensu alerts that flow through to PagerDuty and actually wake people up if necessary. This empowers us to respond quickly to customer issues. However for false positives it’s really frustrating to get a phone call for it. Even worse, false positives train team members to start ignoring alerts and not to worry about them. Then the presence of alerts becomes normal and it’s barely more useful than not having any alerts.

Solution

Ensure that your alerts are meaningful and actionable. Follow the steps below.

1. Export data from PagerDuty
   1. Click Analytics
   2. Click Incident Volume

2. From there you can download csv’s to get your data

3. Concat all the csv’s so you have 6 months of data

cat *csv > combined.csv

4. Grep for your service

grep "my-service" combined.csv > filtered_combined.csv

5. Upload the filtered_combined to Google Sheets
6. Get the average time to resolve
7. Hide any that aren’t relevant so they’re omitted
8. Then use that information to adjust occurrences and interval in sensu checks.
9. You’ll find this configuration in /etc/sensu/conf.d/checks/my_check.json

"occurrences": 3,

10. Additionally you can configure the warning and critical levels for many sensu plugins
11. Take for example the below check with a warning of 80% disk usage and a critical of 90% disk usage

{
     "checks": {
        "check-disk-usage-linux": {
"handlers": ["mailer"],
      "command": "/opt/sensu/embedded/bin/check-disk-usage.rb -w 80 -c 90",
      "interval": 60,
      "occurrences": 5,
      "subscribers": [ "linux" ]
        }
      }
    }

References

1. Merging contents of multiple .csv files into single .csv file
2. Step By Step: Install and Configure Sensu + Grafana

Bacula 101: Is it running?

Problem

You’ve setup and configured (or inherited) a bacula installation on CentOS. It seems to be working and not starting fires. How can you check that everything is running as expected?

Solution

1. Look no further than our trusty service command

service --status-all

• It will provide the following output.

bacula-dir (pid 2868) is running...
bacula-fd (pid 2881) is running...
bacula-sd (pid 2894) is running...

Bacula-dir

This stands for Bacula’s director. As the name implies, this service coordinates all backup and restore jobs that you’ve configured. It’s critical that it can reach out to the below services, which may or may not be co-located.

Bacula-fd

This stands for Bacula file daemon (or service). It runs locally on the server and executes the backup or restore commands issued to it by the director.

Bacula-sd

This stands for Bacula storage daemon (or service). It assists in maintaining, archiving and removing volumes that are used for backups.

References

1. What is Bacula?
2. Bacula CheatSheet

Bacula – Filesets precedence

Problem

You’re configuring bacula but notice that there are multiple places to configure a Job and it’s properties. The challenge comes in creating the job or making the appropriate changes to FileSets without experiencing unintended consequences.

Solution

JobDefs = JobDefs-Resource-Name If a JobDefs-Resource-Name is specified, all the values contained in the named JobDefs resource will be used as the defaults for the current Job. Any value that you explicitly define in the current Job resource, will override any defaults specified in the JobDefs resource. The use of this directive permits writing much more compact Job resources where the bulk of the directives are defined in one or more JobDefs. This is particularly useful if you have many similar Jobs but with minor variations such as different Clients. A simple example of the use of JobDefs is provided in the default bacula-dir.conf file.

It’s important to remember the architecture of these two concepts. JobDefs were introduced as a way to template and avoid duplication. However every property can be overridden in the Job definition even if it exists in the JobDef definition. This is true for FileSets as well as any property.

References

1. Bacula Manual: Configuration the Director

Mongo Gotcha – DBHash Edition

Problem

You need to run dbhash against your Mongo database. However it doesn’t work when you try the below command that leverages use myDB in a call to Mongo from the command line the way you can in the cli.

	..[ryan@host] – [~/Documents/git/repo] – [Mon Aug 26, 12:16]
	..[$] <( (git)-[branch]-)> mongo
	MongoDB shell version v4.0.3
	connecting to: mongodb://127.0.0.1:27017
	WARNING: No implicit session: Logical Sessions are only supported on server versions 3.6 and greater.
	Implicit session: dummy session
	MongoDB server version: 3.2.15
	WARNING: shell and server versions do not match
	> use myDB
	switched to db myDB
	> db.runCommand({ dbHash:1 })
	{
	"host" : "fa34c6baf5a9",
	"collections" : {
	"3_20_1_1" : "b21dc0787d226ca7668f901d18f434f8",
	"3_20_2_1" : "4f60ad3345ef546712ec3e72fc86f95b",
	"3_5_3" : "da5097b8683c8f19c48c22abf63ebffa",
	"3_5_5" : "4121a956b7160d49ca0ca8d6bf4b282a",
	"4_20_3_2" : "573da6f1fac32422403a2e2a1c7b16e6",
	"4_5_7" : "25951ce17fbc465187c7c6e075846686",
	"5_20_4_3" : "4057fa6630cd40e474179ea5a9c4d87b",
	"5_5_9" : "39cbf0af85f1e486cdbf8e7fc75e06e4"
	},
	"md5" : "bc73814f993c792129acd5ad3c5565a0",
	"timeMillis" : 76,
	"fromCache" : [ ],
	"ok" : 1
	}

view raw output.txt hosted with ❤ by GitHub

	..[ryan@host] – [~/Documents/git/repo] – [Mon Aug 26, 12:19]
	..[$] <( (git)-[branch]-) > mongo –host=localhost –port=27017 –eval 'use myDB; db.runCommand( { dbHash: 1 } )'
	MongoDB shell version v4.0.3
	connecting to: mongodb://localhost:27017/
	WARNING: No implicit session: Logical Sessions are only supported on server versions 3.6 and greater.
	Implicit session: dummy session
	MongoDB server version: 3.2.15
	WARNING: shell and server versions do not match
	2019-08-26T12:19:38.598-0400 E QUERY [js] SyntaxError: missing ; before statement @(shell eval):1:4

view raw output.txt hosted with ❤ by GitHub

Solution

I needed to write db = db.getSiblingDB(‘myDB’); instead of use myDB; to make Mongo happy on the command line. You can see a complete example below.

	..[ryan@host] – [~/Documents/git/repo] – [Mon Aug 26, 12:19]
	..[$] <( (git)-[branch]-)> mongo –host=localhost –port=27017 –eval "db = db.getSiblingDB(‘myDB’); db.runCommand( { dbHash: 1 } )"
	MongoDB shell version v4.0.3
	connecting to: mongodb://localhost:27017/
	WARNING: No implicit session: Logical Sessions are only supported on server versions 3.6 and greater.
	Implicit session: dummy session
	MongoDB server version: 3.2.15
	WARNING: shell and server versions do not match
	{
	"host" : "fa34c6baf5a9",
	"collections" : {
	"3_20_1_1" : "b21dc0787d226ca7668f901d18f434f8",
	"3_20_2_1" : "4f60ad3345ef546712ec3e72fc86f95b",
	"3_5_3" : "da5097b8683c8f19c48c22abf63ebffa",
	"3_5_5" : "4121a956b7160d49ca0ca8d6bf4b282a",
	"4_20_3_2" : "573da6f1fac32422403a2e2a1c7b16e6",
	"4_5_7" : "25951ce17fbc465187c7c6e075846686",
	"5_20_4_3" : "4057fa6630cd40e474179ea5a9c4d87b",
	"5_5_9" : "39cbf0af85f1e486cdbf8e7fc75e06e4"
	},
	"md5" : "bc73814f993c792129acd5ad3c5565a0",
	"timeMillis" : 86,
	"fromCache" : [ ],
	"ok" : 1
	}

view raw output.txt hosted with ❤ by GitHub

References

1. Mongo DB Docs: write scripts for the mongo shell
2. StackOverflow: How to execute mongo commands through shell scripts?

Save Command To File

Problem

You would like to know what input generated that output. It may be for compliance reasons and providing evidence. It may just be for your own sanity. It’s nice to be able to have both.

Solution

(set -x; curl https://mysite.com/test_frontend; echo "\n"; curl https://mysite.com/test_backend; \
    echo "\n"; ) 2>&1 | tee ~/${HOSTNAME}_tests.txt

1. The set -x portion will echo back the commands to your output
2. The echo "\n"; adds a line to the output to make it more readable
3. 2>&1 pipes stderr and output to the same output stream
4. | tee writes the output to both the console and the file

References

1. AskUbuntu: How do I save terminal output to a file?