# kubecon2020

**Repository Path**: nats-io/kubecon2020

## Basic Information

- **Project Name**: kubecon2020
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Apache-2.0
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2022-07-25
- **Last Updated**: 2025-09-17

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

#+TITLE: NATS Streams and Services: From Zero to Hero

/KubeCon 2020: NATS Streams and Services: From Zero to Hero/

Talk: https://sched.co/ekHe

[[https://nats.io][NATS]] is high performance cloud native messaging system that allows you to build globally available and secure applications based on streams and services that are both fast and simple to operate. In this talk you will learn: how to get started with NATS streams and services, about the *NATS Adaptive Edge Architecture* to span clusters across different networks and clusters using leafnodes and gateway connections, followed by a deep dive into the latest NATS features and how to deploy in Kubernetes with Helm.

*You can watch the tutorial by clicking on the image below*:

[[https://www.youtube.com/watch?v=3KxjTWcy91o][https://img.youtube.com/vi/3KxjTWcy91o/maxresdefault.jpg]]

* Checkout the repo

Follow along with this repo: https://github.com/nats-io/kubecon2020

#+begin_src sh
git clone https://github.com/nats-io/kubecon2020
#+end_src

* Setup decentralized auth

** Installing NSC

#+begin_src
curl -LO https://raw.githubusercontent.com/nats-io/nsc/master/install.sh
less install.sh
sh ./install.sh
#+end_src

** Initialize NSC

#+begin_src sh
curl -fSl https://nats-io.github.io/k8s/setup/nsc-setup.sh | sh
source .nsc.env
#+end_src

** Inspect NSC objects

#+begin_src sh
tree nsc/ | less
nsc describe jwt -f nsc/accounts/nats/KO/KO.jwt
#+end_src

** Creating the accounts

- We will separate the system into two accounts:
  + An ADMIN account for management of users (create/revoke)
  + An CHAT account for the users to interact with each other.

- We need 3 users at least:
  + Admin User          :: Credentials Provisioner and revocation
  + Chat User           :: Dynamically generated
  + Bootstrap User      :: Chat Credentials Requestor

#+begin_src sh
nsc add account --name CHAT
nsc add account --name ADMIN
nsc list accounts
nsc describe jwt -f ./nsc/accounts/nats/KO/accounts/CHAT/CHAT.jwt
nsc describe jwt -f ./nsc/accounts/nats/KO/accounts/ADMIN/ADMIN.jwt
#+end_src

** Add a new signing key for CHAT and ADMIN accounts

This is needed to be able to create users dynamically by the credentials provisioner.

#+begin_src sh
$ nsc generate nkey --account CHAT --store
ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH
account key stored /Users/wallyqs/repos/nats-dev/src/github.com/nats-io/kubecon2020/nsc/nkeys/keys/A/DC/ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH.nk

#
# This signing key will be used to create users
#
$ nsc edit account -n CHAT --sk AAT6ZEPUN4QIP75K572IRETRJQJP2NJ7LU3FPHTNYSHDBUA4U5NOXNPQ
[ OK ] added signing key "ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH"
[ OK ] edited account "CHAT"

$ nsc generate nkey --account ADMIN --store
AAEYWYQI7LICX2ZRIGVDKJYEAV7V3YK6BEDVDOBOVSGBJT3NB24QMIYN
account key stored /Users/wallyqs/repos/nats-dev/src/github.com/nats-io/kubecon2020/nsc/nkeys/keys/A/AE/AAEYWYQI7LICX2ZRIGVDKJYEAV7V3YK6BEDVDOBOVSGBJT3NB24QMIYN.nk

$ nsc edit account -n ADMIN --sk AD6Q2YIB5YETBUHTB72IMB4KCQI2YPRENU4A6LD6WGITKWK6BBSBV6UT
[ OK ] added signing key "AAEYWYQI7LICX2ZRIGVDKJYEAV7V3YK6BEDVDOBOVSGBJT3NB24QMIYN"
[ OK ] edited account "ADMIN"

$ nsc describe jwt -f ./nsc/accounts/nats/KO/accounts/CHAT/CHAT.jwt | grep Signing
│ Signing Keys              │ ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH │

$ nsc describe jwt -f ./nsc/accounts/nats/KO/accounts/ADMIN/ADMIN.jwt | grep Signing
│ Signing Keys              │ AAEYWYQI7LICX2ZRIGVDKJYEAV7V3YK6BEDVDOBOVSGBJT3NB24QMIYN │
#+end_src

** Create user for the credentials provisioner

This is in the ADMIN account, a user that is able to provision credentials.

#+begin_src
$ nsc add user -a ADMIN chat-access \
   -K $NKEYS_PATH/keys/A/D6/AD6Q2YIB5YETBUHTB72IMB4KCQI2YPRENU4A6LD6WGITKWK6BBSBV6UT.nk \
   --allow-sub 'chat.req.access' \
   --allow-sub 'chat.KUBECON.online' \
   --allow-pubsub 'chat.req.provisioned' \
   --allow-pubsub 'chat.req.provisioned.updates' \
   --allow-pubsub 'chat.req.revoke' \
   --allow-pubsub '_INBOX.>' \
   --allow-pubsub '_R_.>' \
   --allow-pub-response
[ OK ] generated and stored user key "UALGE5RGSDC3DC4DWIO2KU6Y2R35F6UWKD4GGDBL3TTMXROPKKAPRZBQ"
[ OK ] generated user creds file `~/repos/nats-dev/src/github.com/nats-io/kubecon2020/nsc/nkeys/creds/KO/ADMIN/chat-access.creds`
[ OK ] added user "chat-access" to account "ADMIN"

$ nsc describe jwt -f $NKEYS_PATH/creds/KO/ADMIN/chat-access.creds
#+end_src

** Create user for the credentials request

This is a bootstrapping user shared by everyone, like a guest user.

#+begin_src
$ nsc add user -a ADMIN chat-creds-request \
   -K $NKEYS_PATH/keys/A/D6/AD6Q2YIB5YETBUHTB72IMB4KCQI2YPRENU4A6LD6WGITKWK6BBSBV6UT.nk \
   --allow-pubsub '_INBOX.>' \
   --allow-pubsub '_R_.>' \
   --allow-pub 'chat.req.access'

$ nsc describe jwt -f $NKEYS_PATH/creds/KO/ADMIN/chat-creds-request.creds
#+end_src

** Add the export and import

#+begin_src
nsc add export -a ADMIN --service -n chat-access  -s chat.req.access
nsc add export -a CHAT  -n chat-online  -s chat.KUBECON.online

rm ./nsc/accounts/nsc.json

nsc add import -a CHAT  --service --src-account $(nsc list accounts 2>&1 | grep ADMIN | awk '{print $4}') -n chat-access --remote-subject chat.req.access   -s chat.req.access
nsc add import -a ADMIN -n chat-online \
    --src-account $(nsc list accounts 2>&1 | grep CHAT | awk '{print $4}') \
    --remote-subject chat.KUBECON.online
#+end_src

** Setting up the NATS Server locally

Generate the NATS configuration.

#+begin_src sh :results output
source .nsc.env
nsc list accounts
mkdir conf jwt
nsc generate config --sys-account SYS --nats-resolver > conf/resolver.conf
cat conf/resolver.conf
#+end_src

Start the NATS Server:

#+begin_src sh
nats-server -c conf/resolver.conf
#+end_src

** Smoke test

Create a mock admin responder:

#+begin_src sh
nats-rply -creds $NKEYS_PATH/creds/KO/ADMIN/chat-access.creds chat.req.access example
#+end_src

Try to make a request:

#+begin_src sh
nats-req -creds $NKEYS_PATH/creds/KO/ADMIN/chat-creds-request.creds chat.req.access example
#+end_src

* COMMENT Using the Chat Application

** Running the provisioner

Run the provisioner with the public JWT of the CHAT account and the signing key
to create users under the CHAT account.

#+begin_src
go run main.go --acc $NSC_HOME/nats/KO/accounts/CHAT/CHAT.jwt \
    --sk $NKEYS_PATH/keys/A/DC/ADCQ6MQXSOSLMFIW7J6EDLJO2NCQIH4C44YR3MOWS6QLQYQFDV2UXBHH.nk \
    --creds $NKEYS_PATH/creds/KO/ADMIN/chat-access.creds
    --osk $NKEYS_PATH/keys/O/AQ/OAQFB4CLUH2SZR7HOQAV6E7G4INPW36S7YBB5KLUZ3ABLWPWJ4FTRRTA.nk \
    --syscreds $NKEYS_PATH/creds/KO/SYS/sys.creds
#+end_src

** Getting some credentials

#+begin_src
nats-req -creds nsc/nkeys/creds/KO/ADMIN/chat-creds-request.creds chat.req.access wallyqs 2> my.creds
#+end_src

** Starting the Chat app

#+begin_src sh
cd ./chat
go build
./chat --creds ../my.creds
#+end_src

** Revoking a user

To revoke:

#+begin_src sh
nsc revocations add_user -a ADMIN -u UCQYJLDPMVWHETFPF3ZT5DKCVAKGPX33KEZC6DZUHL3DB3VBYNJATOQQ
nats-req -creds $NKEYS_PATH/creds/KO/SYS/sys.creds "\$SYS.REQ.ACCOUNT.$(nsc list accounts 2>&1 | grep ADMIN   | awk '{print $4}').CLAIMS.UPDATE" $(cat $NSC_HOME/nats/KO/accounts/ADMIN/ADMIN.jwt)
#+end_src

* Deploying to K8S: Infra setup

** Creating K8S clusters for NATS

You can find info here:

https://docs.nats.io/nats-on-kubernetes/super-cluster-on-digital-ocean

Let's create a cluster in Digital Ocean:

#+begin_src sh
doctl kubernetes cluster create nats-k8s-sfo2 --count 3 --region sfo2
#+end_src

** Open up the firewall for the required ports

- 4222 is the client port
- 7422 is the port for leafnodes
- 7522 is the port for gateway connections (cluster of clusters)
- 443 is the websocket port

#+begin_src sh
for firewall in `doctl compute firewall list | tail -n 3 | awk '{print $1}'`; do
  doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:4222,address:0.0.0.0/0
  doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7422,address:0.0.0.0/0
  doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:7522,address:0.0.0.0/0
  doctl compute firewall add-rules $firewall --inbound-rules protocol:tcp,ports:443,address:0.0.0.0/0  
done
#+end_src

* Setting up external access for the NATS cluster

** Deploying external-dns

First deploy the external-dns component using the DigitalOcean provider:

#+BEGIN_SRC sh
echo '
rbac:
  create: true

provider: digitalocean

digitalocean:
  apiToken: himitsu

interval: "1m"

policy: sync # or upsert-only
' > k8s/external-dns.yaml

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install external-dns bitnami/external-dns -f k8s/external-dns.yaml
#+END_SRC

** Create NodePort service to create A records for each server

This =NodePort= is required in order to be able to expose the host ports.

#+BEGIN_SRC yaml
apiVersion: v1
kind: Service
metadata:
  name: nats-nodeport
  labels:
    app: nats
  annotations:
    external-dns.alpha.kubernetes.io/hostname: sfo.nats.chat
spec:
  type: NodePort
  selector:
    app: nats
  externalTrafficPolicy: Local
  ports:
  - name: client
    port: 4222
    nodePort: 30222
    targetPort: 4222
  - name: websocket
    port: 443
    nodePort: 30223
    targetPort: 443
#+END_SRC

#+begin_src 
kubectl apply -f k8s/node-port.yaml
#+end_src

This will make it possible to reach out to the NATS cluster by using the =sfo.nats.chat= domain:

#+BEGIN_SRC sh :results output
nslookup sfo.nats.chat
#+END_SRC

** Create load balancer for the websockets port

#+begin_src yaml
apiVersion: v1
kind: Service
metadata:
  name: nats-lb
spec:
  type: LoadBalancer
  selector:
    app: nats-chat-frontend
  ports:
    # - protocol: TCP
    #   port: 4222
    #   targetPort: 4222
    #   name: client
    - protocol: TCP
      port: 80
      targetPort: 8080
      name: websocket
#+end_src

* Deploying NATS to a K8S Cluster

** Add Helm NATS repos

#+begin_src
brew install helm
helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update
#+end_src

** Upload the NATS Accounts bootstrap file

#+begin_src sh
kubectl --context do-sfo2-nats-k8s-sfo2 create cm nats-accounts --from-file conf/resolver.conf
#+end_src

** Setup TLS

We will need TLS for the websockets:

#+begin_src 
kubectl create secret generic nats-tls \
        --from-file=letsencrypt/live/sfo.nats.chat/fullchain.pem \
        --from-file=letsencrypt/live/sfo.nats.chat/privkey.pem

kubectl create secret generic nats-frontend-tls \
        --from-file=letsencrypt/live/nats.chat/fullchain.pem \
        --from-file=letsencrypt/live/nats.chat/privkey.pem
#+end_src

** Create the secrets

Secrets for the provisioner:

#+begin_src 
mkdir creds
cp ./nsc/nkeys/keys/A/AT/AAT6ZEPUN4QIP75K572IRETRJQJP2NJ7LU3FPHTNYSHDBUA4U5NOXNPQ.nk creds/sk.nk 
cp ./nsc/nkeys/keys/O/C2/OC22KWMRNWFF4RAENUOU6DAH4LBEZSKEXMVA57ZQLTP3HJ37IELNV3FO.nk creds/osk.nk
kubectl create secret generic nats-admin-creds \
        --from-file=./nsc/accounts/nats/KO/accounts/CHAT/CHAT.jwt \
        --from-file=./creds/osk.nk \
        --from-file=./creds/sk.nk \
        --from-file=./nsc/nkeys/creds/KO/SYS/sys.creds \
        --from-file=./nsc/nkeys/creds/KO/ADMIN/chat-access.creds
#+end_src

Generic bootstrap credentials for users:

#+begin_src sh
kubectl create secret generic nats-bootstrap-creds --from-file=bootstrap-creds=./nsc/nkeys/creds/KO/ADMIN/chat-creds-request.creds
#+end_src

* Create the NATS Cluster

#+begin_src sh
# helm install nats nats/nats -f k8s/sfo-nats-server.yaml
helm install nats ./helm/helm/charts/nats -f k8s/sfo-nats-server.yaml
#+end_src

Upload the accounts:

#+begin_src sh
nats-req -s tls://sfo.nats.chat:4222 -creds $NKEYS_PATH/creds/KO/SYS/sys.creds "\$SYS.REQ.ACCOUNT.$(nsc list accounts 2>&1 | grep CHAT  | awk '{print $4}').CLAIMS.UPDATE" $(cat ./nsc/accounts/nats/KO/accounts/CHAT/CHAT.jwt)

# FIXME: workaround to prevent colors matching in the line below.
rm ./nsc/accounts/nsc.json

nats-req -s tls://sfo.nats.chat:4222 -creds $NKEYS_PATH/creds/KO/SYS/sys.creds "\$SYS.REQ.ACCOUNT.$(nsc list accounts 2>&1 | grep ADMIN | awk '{print $4}').CLAIMS.UPDATE" $(cat ./nsc/accounts/nats/KO/accounts/ADMIN/ADMIN.jwt)
#+end_src

** Deploy the applications

#+begin_src sh
kubectl apply -f k8s/creds-provisioner.yaml
kubectl apply -f k8s/chat-frontend-deploy.yaml
#+end_src

* Get some credentials

#+begin_src 
nats-req -s tls://sfo.nats.chat:4222 --creds ./nsc/nkeys/creds/KO/ADMIN/chat-creds-request.creds chat.req.access wallyqs 2> my.creds  
#+end_src

* Start the NATS Chat

#+begin_src 
nats-req -s tls://sfo.nats.chat:4222 --creds ../nsc/nkeys/creds/KO/ADMIN/chat-creds-request.creds chat.req.access wallyqs3 2> chat.creds  
#+end_src