Posted on by Ryan

AWS MSK Create & List Topics

A close up of an old typewriter keyboard.

Problem

I needed to created topics in Amazon Web Services(AWS) Managed Streaming for Apache Kafka(MSK) and I wanted to list out the topics after they were created to verify.

Solution

This solution is written in python using the confluent-kafka package. It connects to the Kafka cluster and adds the new topics. Then it prints out all of the topics for verification

from confluent_kafka.admin import AdminClient, NewTopic
import os
a = AdminClient(
{
'bootstrap.servers': os.environ.get("AWS_KAFKA_BOOTSTRAP_SERVERS"),
'security.protocol': 'SASL_SSL',
'sasl.mechanisms': 'AWS_MSK_IAM',
'sasl.aws.access.key.id': os.environ.get("AWS_ACCESS_KEY_ID"),
'sasl.aws.secret.access.key': os.environ.get("AWS_SECRET_ACCESS_KEY"),
'sasl.aws.security.token': os.environ.get("AWS_SESSION_TOKEN"),
'sasl.aws.region': 'us-east-1',
'ssl.ca.location': '/etc/ssl/certs/ca-certificates.crt',
}
)
new_topics = [NewTopic(topic, num_partitions=3) for topic in ["topic1", "topic2"]]
# Note: In a multi-cluster production scenario, it is more typical to use a replication_factor of 3 for durability.
# Call create_topics to asynchronously create topics. A dict
# of <topic,future> is returned.
fs = a.create_topics(new_topics)
# Wait for each operation to finish.
for topic, f in fs.items():
try:
f.result() # The result itself is None
print("Topic {} created".format(topic))
except Exception as e:
print("Failed to create topic {}: {}".format(topic, e))
topics = a.list_topics().topics
print("There are currently {} topics. They are {}".format(len(topics), topics))
view raw add_topics.py hosted with ❤ by GitHub

References

  1. How to create topic in Kafka with Python-kafka admin client?
  2. Pip package confluent-kafka
Posted on by Ryan

Generate all PlantUML diagrams

Problem

You have many C4 PlantUML diagrams in nested folders that you would like to render without having to run the PlantUML cli on each one.

Solution

find . -type f -iname "*.puml" -exec echo "Generating plantuml diagram for {}" \; -exec plantuml {} \;
view raw command.bash hosted with ❤ by GitHub
  • The output will look like the below
┌─[user@laptop] – [~/my-project] – [Fri Feb 16, 16:58]
└─[$] <git:(feature-branch)> find . -type f -iname "*.puml" -exec echo "Generating plantuml diagram for {}" \; -exec plantuml {} \;
Generating plantuml diagram for ./project1/footer.puml
Generating plantuml diagram for ./project1/1 Context.puml
Generating plantuml diagram for ./project1/header.puml
Generating plantuml diagram for ./project1/2 Container.puml
Generating plantuml diagram for ./project2/footer.puml
Generating plantuml diagram for ./project2/1 Context.puml
Generating plantuml diagram for ./project2/header.puml
Generating plantuml diagram for ./project3/network.puml
Generating plantuml diagram for ./project4/1 Context.puml
Generating plantuml diagram for ./project4/2 Container.puml
Generating plantuml diagram for ./project5/Pilot/1 Context.puml
Generating plantuml diagram for ./project5/Pilot/2 Network Container.puml
Generating plantuml diagram for ./project5/Pilot/2 Container.puml
Generating plantuml diagram for ./project5/Pilot/definitions.puml
Generating plantuml diagram for ./project6/1 Context.puml
Generating plantuml diagram for ./project6/2 Container.puml
Generating plantuml diagram for ./project6/definitions.puml
find . -type f -iname "*.puml" -exec echo "Generating plantuml diagram for {}
Time: 33.93s user | 2.96s system | 112% cpu | 32.708 total | 336528 KiB max RSS
view raw output.bash hosted with ❤ by GitHub

References

  1. Manual Page for find
  2. Github: C4 Plant UML
Posted on by Ryan · 1 Comment

Install PlantUML on Mac

Problem

You would like to create PlantUML diagrams on your Mac but don’t have the prerequisites or cli installed.

Solution

Steps

  1. Install java with brew install java
  2. Install PlantUML with brew install plantuml

Output for Java

  • Your output will look like the below if it is a new installation
┌─[user@laptop] – [~] – [Wed Feb 14, 15:13]
└─[$] <> brew install java
==> Downloading https://formulae.brew.sh/api/formula.jws.json
##O=- # #
==> Downloading https://formulae.brew.sh/api/cask.jws.json
############################################################################################################################################################################################################# 100.0%
Warning: Cask homebrew/cask/java was renamed to homebrew/core/java.
==> Downloading https://ghcr.io/v2/homebrew/core/openjdk/manifests/21.0.2
Already downloaded: /Users/user/Library/Caches/Homebrew/downloads/d437bb150fa297f0ee7f7f26594cb0a1e7aec55a45ec6570ed8660a033b7c3f9–openjdk-21.0.2.bottle_manifest.json
==> Fetching openjdk
==> Downloading https://ghcr.io/v2/homebrew/core/openjdk/blobs/sha256:9850be1875b9df8e9fa3510b6f2e947be2ff228d64a1c8e0daebc57a018ce2ef
Already downloaded: /Users/user/Library/Caches/Homebrew/downloads/ea1872f4761168acb4af2804e61d03cc5b31765cbc2ee364efcf7e10a67d2c8d–openjdk–21.0.2.arm64_sonoma.bottle.tar.gz
==> Pouring openjdk–21.0.2.arm64_sonoma.bottle.tar.gz
==> Caveats
For the system Java wrappers to find this JDK, symlink it with
sudo ln -sfn /opt/homebrew/opt/openjdk/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk.jdk
openjdk is keg-only, which means it was not symlinked into /opt/homebrew,
because macOS provides similar software and installing this software in
parallel can cause all kinds of trouble.
If you need to have openjdk first in your PATH, run:
echo 'export PATH="/opt/homebrew/opt/openjdk/bin:$PATH"' >> ~/.zshrc
For compilers to find openjdk you may need to set:
export CPPFLAGS="-I/opt/homebrew/opt/openjdk/include"
==> Summary
🍺 /opt/homebrew/Cellar/openjdk/21.0.2: 600 files, 331.2MB
==> Running `brew cleanup openjdk`…
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
view raw output.sh hosted with ❤ by GitHub
  • Your output will look like the below if it is already installed
┌─[user@laptop] – [~] – [Wed Feb 14, 14:55]
└─[$] <> brew install java
==> Downloading https://formulae.brew.sh/api/formula.jws.json
############################################################################################################################################################################################################# 100.0%
==> Downloading https://formulae.brew.sh/api/cask.jws.json
############################################################################################################################################################################################################# 100.0%
Warning: Cask homebrew/cask/java was renamed to homebrew/core/java.
Warning: openjdk 21.0.2 is already installed and up-to-date.
To reinstall 21.0.2, run:
brew reinstall openjdk
view raw output.sh hosted with ❤ by GitHub

Output for PlantUML

  • Your output will look like the below if it is a new installation
┌─[user@laptop] – [~] – [Wed Feb 14, 15:16]
└─[$] <> brew install plantuml
==> Downloading https://ghcr.io/v2/homebrew/core/plantuml/manifests/1.2024.1
Already downloaded: /Users/user/Library/Caches/Homebrew/downloads/34e1194efa92f43fc1d37f47254e076fc6defb4330b5500d36e5685aed26e3e6–plantuml-1.2024.1.bottle_manifest.json
==> Fetching plantuml
==> Downloading https://ghcr.io/v2/homebrew/core/plantuml/blobs/sha256:3f9404a546ff9e549e1b640e77515b3815819a4b36239fc9b17effe8135c65e3
Already downloaded: /Users/user/Library/Caches/Homebrew/downloads/2d560846774dbc633917be48445efd06524e670aaf0b7299d210b7cff5c91afa–plantuml–1.2024.1.all.bottle.tar.gz
==> Pouring plantuml–1.2024.1.all.bottle.tar.gz
🍺 /opt/homebrew/Cellar/plantuml/1.2024.1: 4 files, 11.3MB
==> Running `brew cleanup plantuml`…
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
view raw output.sh hosted with ❤ by GitHub
  • Your output will look like the below if it is already installed
┌─[user@laptop] – [~] – [Wed Feb 14, 15:19]
└─[$] <> brew install plantuml
Warning: plantuml 1.2024.1 is already installed and up-to-date.
To reinstall 1.2024.1, run:
brew reinstall plantuml
view raw output.sh hosted with ❤ by GitHub

References

  1. Install Homebrew
  2. Homebrew Java
  3. Homebrew PlantUML
  4. PlantUML homepage
Posted on by Ryan

AWS Missing Required Attribute

An empty folding chair sitting in front of a storefront with the sun setting.

Problem

I was trying to create an Elastic Container Service(ECS) service on an ECS cluster that used EC2 instances in AWS. However the tasks would fail to deploy with the following cryptic error.

service my-service was unable to place a task because no container instance met all of its requirements. The closest matching container-instance 1234a5b6cd7890e2f34gh5678i8jk is missing an attribute required by your task. For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.

Solution

You can configure an Amazon ECS service to run on a subnet that’s different from the subnet for the container instance. In this case, the output of the ecs-cli-check-attributes command shows None for missing attributes, even though the task fails with the missing attribute error. Be sure that the subnets for your service and the container instance match. To do this, recreate the Amazon ECS service in the subnet where the container instance exists. For more information, see Task definition parameters and Amazon ECS container agent configuration.

Once I made sure that the create-service call like below used the same subnet and security group I was able to deploy the service.

References

  1. AWS RePost: How do I resolve the “[AWS service] was unable to place a task because no container instance met all of its requirements” error in Amazon ECS?
  2. AWS Github amazon-ecs-cli
Posted on by Ryan

Bash query json

A realistic painting of a nature scene with a pair of boulders and a bridge connecting them. A stream runs between the boulders under the bridge. The trees are green and there is orangish moss on the rocks.

Problem

You want to know all of the applications that your team supports across environments. You are working with an api and it gives you a distinct list of all the apps across all environments.

{
"env1":
{
"apps":
[
"app1",
"app2",
"app3",
"app4"
]
},
"env2":
{
"apps":
[
"app1",
"app2",
"app3",
"app4",
"app5",
"app6"
]
},
"env3":
{
"apps":
[
"app1",
"app2",
"app3",
"app4",
"app6"
]
},
"env4":
{
"apps":
[
"app1",
"app2",
"app3",
"app4",
"app6"
]
},
"env5":
{
"apps":
[
"app1",
"app2",
"app4",
"app6",
"app3"
]
},
"env6":
{
"apps":
[
"app1",
"app2",
"app4",
"app6",
"app3"
]
}
}
view raw output.json hosted with ❤ by GitHub

Solution

Fortunately you can easily read the values into an array in bash with jq and some piping.

read -a APP_NAMES < <(cat environments.json | jq -r '.[].apps' | jq -s 'add' | jq 'unique' | jq '.[]' | tr '\n' ' ' )
view raw solution.sh hosted with ❤ by GitHub

One caveat to note is that if you’re using zsh you will need to use read -A rather than read -a to avoid the following error.

read: bad option: -a

References

  1. StackOverflow: How to use ‘readarray’ in bash to read lines from a file into a 2D array – Stack Overflow
  2. jq
Posted on by Ryan

Vault CLI Create Policy

An old bluish teal fence with some of the paint peeling off in front of some bushes and a grassy area with a blurry table in the distance.

Problem

You need to grant permissions to secret paths in a uniform way for multiple users and/or roles. You may even need to explain the access to someone working in security or compliance. This becomes much more difficult if you are granting access separately to each user and/or role.

Solution

Fortunately HashiCorp Vault provides a policy. With a policy, you can define a set of paths and access to those paths. Here is an example below.

┌─[ryan@host] – [~] – [Thu Jun 22, 14:13]
└─[$] <> cat new-policy.hcl
path "secrets/data/env1/app1/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
path "secrets/data/env2/app2/*" {
capabilities = ["create", "read", "update", "delete", "list"]
}
┌─[ryan@host] – [~] – [Thu Jun 22, 14:13]
└─[$] <> cat new-policy.hcl | vault policy write new-policy –
Success! Uploaded policy: new-policy
view raw create_policy.sh hosted with ❤ by GitHub

Policies in HashiCorp Vault provide a convenient way to grant uniform access to multiple users and/or roles. This helps to provide consistent access to enable people to work and avoids over provisioning access.

References

  1. Vault CLI Documentation for vault policy write
Posted on by Ryan

Terraform staying current

A highly detailed and zoomed in black and white picture of a large snail on a branch with leaves

Problem

Depending on your CI/CD story for terraform and the sensitivity of your environments, you may need to apply terraform changes manually. The challenge that can happen is that changes in terraform may not be applied. This creates a challenge when you do try to apply a change but see historic changes as well that were never applied.

Solution

  1. The first step is to understand what changes may be outstanding across your different stacks.
  2. This solution does assume that you are using terraspace
IFS=$'\n' read -rd '' -A stacks <<<"$(terraspace list -t stack)"
for stack in $stacks; do
echo "Planning ${stack#$prefix}"
output=$(terraspace plan ${stack#$prefix} 2>&1)
echo $(echo $output | grep "Plan:")
done
view raw check_stacks.sh hosted with ❤ by GitHub
  1. This will provide the following output
Planning stack1
Plan: 1 to add, 2 to change, 3 to destroy.
Planning stack2
Plan: 4 to add, 5 to change, 6 to destroy.
view raw output.sh hosted with ❤ by GitHub

Then you can use this information to confirm the changes are expected and desired. Then you can apply them so that future changes will be easier and simpler to manage.

References

  1. terraspace plan
  2. terraspace list
  3. how to grep on a variable
  4. Split string on newline and write it into array using read
  5. Split bash string by newline characters
Posted on by Ryan

Tmux tickets

Lots of different concert tickets with some overlapping

Problems

Like many engineers, I have a support shift. When doing so, much of the work is based around tickets. At the same time, it’s very useful to understand what changes are made for each ticket. This helps for future reference and knowledge sharing as well as understanding changes made if anything goes sideways. The question is how to do this without making it add to the burden of support.

Solution

Tmux helps in multiple ways for this by isolating commands for a given ticket and saving the commands with output to a file. The one caveat is make sure to remove any sensitive information as part of the process.

  1. Start a new session with tmux using the command
    • tmux new -s PROJ-1234
  2. This will isolate any commands for the particular ticket
  3. It will preserve the window if it is closed as well. However it will be lost if you restart or shutdown
  4. When you are done, you can run
    • name=$(tmux display-message -p '#S') && tmux capture-pane -pS - >> ~/Desktop/${name}_commands.txt 2>&1
  5. This will append the output from the session to a file prefixed with the session name.
  6. Make sure to remove any passwords, Personal Identifiable Information(PII) or other sensitive information prior to sharing it.

References

  1. Unix & Linux StackExchange: Write all tmux scrollback to a file
  2. SuperUser StackExchange: Is redirection with `>>` equivalent to `>` when target file doesn’t yet exist?
  3. SuperUser StackExchange: How do I know current tmux session name, by running tmux command
Posted on by Ryan

Pipenv Null Markers

Blank sign held up by two dark grey posts on either side. It is surrounded by a green bush and green tree foliage.

Problem

While trying to use pipenv to manage python dependencies, I noticed that I got the following error when trying to convert the Pipfile.lock to a requirements.txt file.

(my-app) ┌─[ryan@host] – [~/Documents/git/my-repo] – [Wed Oct 05, 14:29]
└─[$] pipfile2req Pipfile.lock > requirements.txt
Traceback (most recent call last):
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/bin/pipfile2req", line 8, in
sys.exit(main())
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/lib/python3.9/site-packages/pipfile2req/init.py", line 112, in main
lines = convert_pipfile_or_lock(
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/lib/python3.9/site-packages/pipfile2req/init.py", line 96, in convert_pipfile_or_lock
lines = [
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/lib/python3.9/site-packages/pipfile2req/init.py", line 97, in
requirement_from_pipfile(name, package, hashes)
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/lib/python3.9/site-packages/pipfile2req/requirements.py", line 10, in requirement_from_pipfile
return Requirement.parse(name, package).as_line(include_hashes)
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/lib/python3.9/site-packages/pipfile2req/requirements.py", line 97, in parse
kwargs["markers"] = _merge_markers(markers)
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/lib/python3.9/site-packages/pipfile2req/requirements.py", line 14, in _merge_markers
result = Marker(markers[0])
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/lib/python3.9/site-packages/packaging/markers.py", line 278, in init
self._markers = _coerce_parse_result(MARKER.parseString(marker))
File "/Users/ryan/.local/share/virtualenvs/my-app-g5SfM111/lib/python3.9/site-packages/pyparsing.py", line 1941, in parseString
instring = instring.expandtabs()
AttributeError: 'NoneType' object has no attribute 'expandtabs'
view raw output.sh hosted with ❤ by GitHub

Upon further inspection, I discovered that in some cases the marker value in the Pipfile.lock was null which caused the pipfile2req program to crash.

Solution

The issue had been pipenv was version 2022.9.20 which introduced the issue. I was able to uninstall using pip uninstall pipenv.

┌─[ryan@host] – [~] – [Wed Oct 05, 16:58]
└─[$] <> pip uninstall pipenv
Found existing installation: pipenv 2022.9.20
Uninstalling pipenv-2022.9.20:
Would remove:
/Users/ryan/.pyenv/versions/3.9.6/bin/pipenv
/Users/ryan/.pyenv/versions/3.9.6/bin/pipenv-resolver
/Users/ryan/.pyenv/versions/3.9.6/lib/python3.9/site-packages/pipenv-2022.9.20.dist-info/*
/Users/ryan/.pyenv/versions/3.9.6/lib/python3.9/site-packages/pipenv/*
Proceed (Y/n)? Y
Successfully uninstalled pipenv-2022.9.20
view raw uninstall_pipenv.sh hosted with ❤ by GitHub

Then I re-installed pipenv with the latest version(2022.10.4 at the time of this writing).

┌─[ryan@host] – [~] – [Wed Oct 05, 16:58]
└─[$] <> pip install pipenv
Collecting pipenv
Downloading pipenv-2022.10.4-py2.py3-none-any.whl (3.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 8.9 MB/s eta 0:00:00
Requirement already satisfied: virtualenv-clone>=0.2.5 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from pipenv) (0.5.7)
Requirement already satisfied: setuptools>=36.2.1 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from pipenv) (57.5.0)
Requirement already satisfied: certifi in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from pipenv) (2022.6.15)
Requirement already satisfied: virtualenv in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from pipenv) (20.15.0)
Requirement already satisfied: filelock<4,>=3.2 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from virtualenv->pipenv) (3.7.1)
Requirement already satisfied: platformdirs<3,>=2 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from virtualenv->pipenv) (2.5.2)
Requirement already satisfied: distlib<1,>=0.3.1 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from virtualenv->pipenv) (0.3.4)
Requirement already satisfied: six<2,>=1.9.0 in ./.pyenv/versions/3.9.6/lib/python3.9/site-packages (from virtualenv->pipenv) (1.16.0)
Installing collected packages: pipenv
Successfully installed pipenv-2022.10.4
view raw reinstall_pipenv.sh hosted with ❤ by GitHub

That resolved the issue of putting null markers in the Pipfile.lock file for packages.

ReferenceS

  1. Pipenv lock changes markers in a non deterministic way #4660
  2. package use specified marker in Pipfile may generates error logical condition in Pipfile.lock #3616
  3. pipenv

Posted on by Ryan

Terraform Linting in CI

Trash grabber leaning against a metal trash can on top of grass with trees and shrubs in the background

Problem

Hashicorp Terraform provides a cli with the command terraform fmt which will format terraform files according to the canonical style. This is a great tool. The challenge is that you don’t want to give yourself or team members one more thing they need to remember to do. It would be ideal to integrate it into the Continuous Integration(CI) system so it would always be run but no one would need to remember to do it.

Solution

Fortunately, arguments can be added to provide useful feedback to anyone that is committing changes without them having to run the command themselves. Here’s an example.

..[ryan@server] – [~/my-project] – [Wed May 11, 02:42]
..[$] <( (git)-[main]-)> terraform fmt -check -diff
tf_code.tf
— old/tf_code.tf
+++ new/tf_code.tf
@@ -5,5 +5,5 @@
resource "aws_s3_bucket" "tf_course" {
bucket = "tf-course-20220511"
acl = "private"acl = "private"
}
\ No newline at end of file
view raw solution.sh hosted with ❤ by GitHub

This can be incorporated into your CI system that can be run on each commit and it is quick as well(terraform fmt -check -diff 0.03s user 0.04s system 34% cpu 0.200 total).

References

  1. DevOps StackExchange: Is there a way to lint Terraform?
  2. StackOverflow: Terraform fails without any error message
  3. Terraform Docs fmt command