From 26983b38d1715cf2527f008301e5762e566389f9 Mon Sep 17 00:00:00 2001 From: "YiLin.Li" Date: Tue, 15 Aug 2023 21:01:13 +0800 Subject: [PATCH] practices-keylime.md: remove some unnecessary sections Signed-off-by: YiLin.Li --- src/practices-keylime.md | 643 +++++++++++++++++++++++---------------- 1 file changed, 375 insertions(+), 268 deletions(-) diff --git a/src/practices-keylime.md b/src/practices-keylime.md index e4856b8..324b58f 100644 --- a/src/practices-keylime.md +++ b/src/practices-keylime.md @@ -11,65 +11,40 @@ Keylime 由三个主要组件组成；verifier、registrar和agent。 - registrar是在 Keylime 中注册的所有agent的数据库，并托管 TPM 供应商的公钥。 - agent部署在要监控的TPM机器上 +![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1692101838904-ca289e62-b217-4c33-af50-33e260d96ee5.png) + 此外keylime还提供tenant工具便于用户远程管理agent。 # 龙蜥社区在keylime社区的工作与探索 -龙蜥社区自其可信计算SIG成立以外，一直在关注可信计算业界进展和国际OSV厂商的可信计算方案。同时龙蜥社区在keylime社区积极贡献代码与适配，一共在rust-keylime和keylime两个仓库提交16个patch，包括多个features、bugfixes和文档。详见[keylime release notes](https://github.com/keylime/keylime/releases)和[rust-keylime](https://github.com/keylime/rust-keylime/releases), 具体包括： +龙蜥社区自其可信计算SIG成立以外，一直在关注可信计算业界进展和国际OSV厂商的可信计算方案。同时龙蜥社区在keylime社区积极贡献代码与适配，一共在rust-keylime和keylime两个仓库提交并合入17个patch，包括多个features、bugfixes和文档。详见[keylime release notes](https://github.com/keylime/keylime/releases)和[rust-keylime](https://github.com/keylime/rust-keylime/releases), 具体包括： - features：集成龙蜥anolis以及下游阿里云Alibaba Cloud Linux OS的安装代码、集成阿里云vTPM EK证书、支持keylime安装时选择缺省的监听端口等。 - bugfixes: 修复measure boot时处理部分eventlog出错、修复rust-keylime跟keylime RESTful APIs版本和接口不一致、移除无用的代码等。 - 文档：修复安装文档和实践文档中多处命令错误等。 -在完成keylime的适配和实践后，龙蜥社区也将自己的keylime经验写入到白皮书中。未来，龙蜥社区除了继续加强与keylime社区的沟通和贡献（参与keylime rust化）外，还将结合自己在国密/国产化/机密计算的积累围绕keylime开展一些国密、国产化、机密计算相关的工作，尽情期待。 - -# 龙蜥Anolis OS上keylime实践 -## Anolis OS上keylime安装与配置、运行 -### 安装 - -keylime分为两个代码仓库： -- [keylime](https://github.com/keylime/keylime)：包含除了agent以外的其它keylime组件（verifier，registrar，tenant） -- [rust-keylime](https://github.com/keylime/rust-keylime): 包含keylime的agent组件 - -#### 安装keylime +| 开源软件名称 | 总计commit数量 | 总计修改行数 | +| :- | --: | --: | +| [rust-keylime](https://github.com/keylime/rust-keylime) | 3 | -19/+20 | +| [keylime](https://github.com/keylime/keylime) | 14 | -24/+156 | -根据以下命令在anolis (以anolis 8.8为例) 上安装keylime - -```shell -yum install -y git jq -git clone https://github.com/keylime/keylime.git -cd keylime -./installer.sh -i -``` +在完成keylime的适配和实践后，龙蜥社区也将自己的keylime经验写入到白皮书中。未来，龙蜥社区除了继续加强与keylime社区的沟通和贡献（参与keylime rust化）外，还将结合自己在国密/国产化/机密计算的积累围绕keylime开展一些国密、国产化、机密计算相关的工作，尽情期待。 -#### 安装rust-keylime +# 龙蜥Anolis OS上keylime用途与实践 -根据以下命令在anolis（以anolis 8.8为例）上安装rust-keylime（keylime agent） +Keylime可以借助PCR或者Measure boot监控远程机器（Agent部署机器）的启动时的状态（完整性等）和借助IMA来监控运行时的完整性。开启对应的策略（policy）后，时刻轮询监控着对应agent的状态，如果发现异常则返回给verifier执行对应的操作（标记失败/停止轮询/打印错误等）。关于这部分的用法详见下文`Anolis OS上keylime高级功能实践`章节。 -1. 安装tpm2-tss软件包(**如果keylime agent跟keylime 其它组件安装在一台机器，则这一步可以省略**) +Keylime也可以通过RESTful APIs去监控/管理/查询 Keylime的agent、verifier以及registrar，便于用户以及运维人员有效的管理Keylime的各个组件以及验证远程机器的完整性等。此外keylime提供更加安全的mtls协议和基于Https的RESTful APIs。其中如果被正确执行，则RESTful APIs返回对应的信息且状态码为200，否则则为错误运行。关于这些APIs的用法详见下文`用Restful API去监控/管理Anolis OS上的各个keylime组件`章节。 -```shell -yum install -y git openssl-devel json-c-devel libcurl-devel libuuid-devel m4 libtool automake autoconf autoconf-archive -git clone https://github.com/tpm2-software/tpm2-tss.git -pushd tpm2-tss -./bootstrap -./configure --prefix=/usr -make -sudo make install -popd -``` +## Anolis OS上keylime安装与配置、运行 +### 安装（以anolis 8.8为例） -2. 安装tpm2-tools软件包(**如果keylime agent跟keylime 其它组件安装在一台机器，则这一步可以省略**) -```shell -git clone https://github.com/tpm2-software/tpm2-tools.git -pushd tpm2-tools -./bootstrap -./configure --prefix=/usr/local -make -sudo make install -popd -``` +keylime分为两个代码仓库： +- [keylime](https://github.com/keylime/keylime)：包含除了agent以外的其它keylime组件（verifier，registrar，tenant）, 下载后执行`cd keylime && ./installer.sh -i` 命令进行安装 +- [rust-keylime](https://github.com/keylime/rust-keylime): 包含keylime的agent组件 + - 安装[tpm2-tss软件包](https://github.com/tpm2-software/tpm2-tss)(**如果keylime agent跟keylime 其它组件安装在一台机器，则这一步可以省略**) + - 安装[tpm2-tools软件包](https://github.com/tpm2-software/tpm2-tools)(**如果keylime agent跟keylime 其它组件安装在一台机器，则这一步可以省略**) + - 安装rust-keylime：可以根据以下命令在anolis（以anolis 8.8为例）上安装rust-keylime（keylime agent） -3. 安装rust-keylime ```shell yum install -y libarchive-devel clang-devel rust cargo openssl-devel jq git clone https://github.com/keylime/rust-keylime.git @@ -78,41 +53,21 @@ cargo build make install useradd keylime mkdir -p /var/lib/keylime/cv_ca -# 将keylime verifier机器上的/var/lib/keylime/cv_ca/cacert.crt拷贝到agent机器上/var/lib/keylime/cv_ca/目录下，以便于后续Agent侧https RESTful APIs的访问 +# 将keylime verifier机器上的/var/lib/keylime/cv_ca/cacert.crt拷贝到agent +# 机器上/var/lib/keylime/cv_ca/目录下，以便于后续Agent侧https RESTful APIs的访问 chown -R keylime /var/lib/keylime ``` ### 配置 -#### verifier配置 - -`/etc/keylime/verifier.conf`为verifier的缺省配置。一般情况下不需要进行修改（当然您也可以根据您的需求进行修改）。 - -#### registrar配置 - -`/etc/keylime/registrar.conf`为registrar的缺省配置。一般情况下不需要进行修改（当然您也可以根据您的需求进行修改）。 - -#### agent配置 - -`/etc/keylime/agent.conf`为agent的缺省配置。当agent跟verifier、registrar部署在同一台机器时，不需要修改agent的配置；否则需要修改agent监听的IP、contact_ip(verifier和tenant用来连接的agent IP)、registrar的IP以便于正确注册和通信。 - -#### tenant配置 - -`/etc/keylime/tenant.conf`为tenant的缺省配置。一般情况下不需要进行修改（当然您也可以根据您的需求进行修改）。 +- verifier配置:`/etc/keylime/verifier.conf`为verifier的缺省配置。一般情况下不需要进行修改（当然您也可以根据您的需求进行修改）。 +- registrar配置:`/etc/keylime/registrar.conf`为registrar的缺省配置。一般情况下不需要进行修改（当然您也可以根据您的需求进行修改）。 +- agent配置:`/etc/keylime/agent.conf`为agent的缺省配置。当agent跟verifier、registrar部署在同一台机器时，不需要修改agent的配置；否则需要修改agent监听的IP、contact_ip(verifier和tenant用来连接的agent IP)、registrar的IP以便于正确注册和通信。 +- tenant配置:`/etc/keylime/tenant.conf`为tenant的缺省配置。一般情况下不需要进行修改（当然您也可以根据您的需求进行修改）。 ### 运行 启动方式： 1. 以二进制方式启动verifier、registrar和agent： -```shell -keylime_verifier & -keylime_registrar & -RUST_LOG=debug keylime_agent & -``` - -- ![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691735004148-a8851d2e-78f7-4bc7-a345-b680ec76c859.png) -- ![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691735017966-82c855d3-d190-4054-b0e4-8b30f38ef1a1.png) -- ![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691740824910-d78537dd-c737-4aa8-9eda-df9aaf8e1e6d.png) - 2. 以systemd方式启动verifier、registrar（具体命令如下） ```shell @@ -123,328 +78,480 @@ systemctl start keylime_registrar systemctl start keylime_agent ``` -- ![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691735031725-7d9744d3-eca7-4902-90d7-d54c49f2d6b5.png) -- ![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691740620033-4ef6d497-edb8-405f-98c9-cc1c59049576.png) - -## 用Restful API去监控/管理Anolis OS上的各个keylime组件 -### registrar - -这里的registrar IP请根据自己的实际IP进行修改 +## Anolis OS上keylime高级功能实践 +### 使用用户选择的PCR进行监控 -#### GET /v2.1/agents/ - -用来获取注册的agents列表，具体命令如下： +该功能需要Agent侧有TPM，但因为TPM的PCR数量有限，扩展性不好。配置tpm_policy并用keylime_tenant工具进行添加，具体命令如下 ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8891/v2.1/agents" | jq . +keylime_tenant -v 121.43.60.253 -t 120.26.100.138 \ + --uuid d432fbb3-d2f1-4a97-9ef7-75bd81c00000 \ + --tpm_policy "{\"15\": [\"0000000000000000000000000000000000000000\", \ + \"0000000000000000000000000000000000000000000000000000000000000000\", \ +\"00000000000000000000000000000000000000000000000000000000000000000000000000\ +0000000000000000000000\"]}" \ + -c add --cert /var/lib/keylime/cv_ca ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691742046552-32684038-4377-408d-a06a-ed8a85534e92.png) - -#### GET /v2.1/agents/{agent_id:UUID} - -获取对应agent的EK证书、 +#### 监控 -命令如下： +成功的case如下（agent时刻监控TPM PCRs的状态） ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8891/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" | jq . + DEBUG keylime_agent::quotes_handler > Calling Integrity Quote with nonce: lCYwDxi2UkTIWM6Fk2aH, mask: 0x408000 + INFO keylime_agent::quotes_handler > GET integrity quote returning 200 response + INFO actix_web::middleware::logger > GET + /v2.1/quotes/integrity?nonce=lCYwDxi2UkTIWM6Fk2aH&mask=0x408000&partial=1&ima_ml_entry=0 + HTTP/1.1 from 121.43.60.253 result 200 (took 1229.894715 ms) + INFO keylime_agent > GET invoked from "121.43.60.253" with uri + /v2.1/quotes/integrity?nonce=WSn8mEpGLjN5I8mhHjPn&mask=0x408000&partial=1&ima_ml_entry=0 ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691742182879-eea791a1-1f94-4baa-aea7-34fb92c08d27.png) +### 使用Measured Boot -#### PUT /v2.1/agents/{agent_id:UUID}/activate +该功能 +- 需要Agent有TPM以及使能IMA +- Keylime提供脚本create_mb_refstate生成对应的measured boot reference state和policy(根据/sys/kernel/security/tpm0/binary_bios_measurements里面的boot event log) +- Keylime提供keylime_tenant（--mb_refstate参数）添加对应的参考值和policy，然后启动Agent -激活agent_id的agent，注意**这是一个http请求，不是https，如果用https会提示这个不是TLS接口** -命令如下: +![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1694077451486-1537c333-48af-4bed-a557-4b802f8fe48c.png) + +第一步，生成measure boot policy ```shell -curl -k -X PUT "http://127.0.0.1:8890/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000/activate" -H 'Content-Type: application/json' -d '{"auth_tag":"166be150040c57b4e2c69ad7a5dd4c57059e5838a1df4715872f6e385e8ce1ed91f22b5e42b46a792718c7cb70f044d5"}' | jq . +cd keylime +./scripts/create_mb_refstate -i /sys/kernel/security/tpm0/binary_bios_measurements \ + ./measured_boot_reference_state.json +cat ./measured_boot_reference_state.json | jq . ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691742275699-4a0668ae-edb5-4a48-a2f4-ea744fb46160.png) - -#### DELETE /v2.1/agents/{agent_id:UUID} - -Remove agent agent_id from registrar - -具体命令如下： +用keylime_tenant工具进行添加mb_reference，对应的命令如下： ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k -X DELETE "https://127.0.0.1:8891/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" | jq . +keylime_tenant -c update -t 120.26.100.138 -v 121.43.60.253 \ + -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000 \ + --mb_refstate ./measured_boot_reference_state.json \ + --cert /var/lib/keylime/cv_ca ``` -此时再查看发现没有该agent了, 使用该命令 +agent侧查看轮询结果:时刻轮询/监控着是否有异常。 ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8891/v2.1/agents" | jq . + INFO keylime_agent > GET invoked from "121.43.60.253" with uri + /v2.1/quotes/integrity?nonce=EiRTFv9tgNBmwy6HwxAC&mask=0xfbff&partial=1&ima_ml_entry=0 + DEBUG keylime_agent::quotes_handler > Calling Integrity Quote with nonce: EiRTFv9tgNBmwy6HwxAC, mask: 0xfbff + INFO keylime_agent::quotes_handler > GET integrity quote returning 200 response + INFO actix_web::middleware::logger > GET + /v2.1/quotes/integrity?nonce=EiRTFv9tgNBmwy6HwxAC&mask=0xfbff&partial=1&ima_ml_entry=0 + HTTP/1.1 from 121.43.60.253 result 200 (took 1452.270707 ms) ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691742405235-772ad1f3-480c-468c-b827-d491ffc6037c.png) +### Runtime Integrity Monitoring -#### POST /v2.1/agents/{agent_id:UUID} +该功能 +- 需要Agent有TPM以及使能IMA +- Keylime提供脚本keylime_create_policy/create_runtime_policy.sh创建对应的runtime policy +- Keylime提供keylime_tenant（--runtime-policy参数）添加对应的runtime policy，然后启动Agent -注册agent_id的agent到registrar.**这是一个http不是https的请求。** -当使用http时 +使用keylime_create_policy工具来生成policy： ```shell -# curl -X POST "http://127.0.0.1:8890/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" -H 'Content-Type: application/json' -d '{ "ekcert": "MIIE3DCCA8SgAwIBAgIBBDANBgkqhkiG9w0BAQsFADBvMQswCQYDVQQGEwJDTjEPMA0GA1UECgwGQWxpeXVuMTIwMAYDVQQLDClBbGl5dW4gVFBNIEVuZG9yc2VtZW50IEtleSBNYW51ZmFjdHVyZSBDQTEbMBkGA1UEAwwSQWxpeXVuIFRQTSBFS01GIENBMB4XDTIzMDUxNzA0MzMxNFoXDTMzMDUxNDA0MzMxNFowaTELMAkGA1UEBhMCQ04xDzANBgNVBAoTBkFsaXl1bjEoMCYGA1UECxMfVFBNIEVuZG9yc2VtZW50IEtleSBDZXJ0aWZpY2F0ZTEfMB0GA1UEAxMWaS1icDE3M3BwZ3dxZmJwd3FxbHdjNDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJ1merAAAU4dbFKmBAwRWTQwVISBekah4sFaT7iBuKJzlylXgAnpgxFzohvu84dbv5MQLYsQ3Au/H2Y7isYCO4U2VJPsKfozZd7FhDvRs4YAQ+ldsQ9128fFZo4OjjZJc27+lR9Gg1CD5oE6vo0XlvqiVJIfG8klnqdIPzqArV1SE3vd5JooZKEpXAcyeZDf7ntkZuL9H38uka2eHwbsQd9A5nfJTPtDw1zMlE8U3Xj8n5IkQV8bOg4U4GFx8tBebXkym90VmCsILfJMisUiuD5uEC/JzxusPUsbBLO1/vL1IAifRwG0i0rT3JJFKeY+BzOErIPLghn/iKWtahlPMOECAwEAAaOCAYcwggGDMBAGA1UdJQQJMAcGBWeBBQgBMFIGA1UdEQEB/wRIMEakRDBCMRYwFAYFZ4EFAgEMC2lkOjAwMDAxMDE0MRAwDgYFZ4EFAgIMBXN3dHBtMRYwFAYFZ4EFAgMMC2lkOjIwMTcwNjE5MAwGA1UdEwEB/wQCMAAwIgYDVR0JBBswGTAXBgVngQUCEDEOMAwMAzIuMAIBAAICAJYwHwYDVR0jBBgwFoAUgaqaZJKI/wcHB1Caz5dO/2ZZUMgwDwYDVR0PAQH/BAUDAwcgADBgBggrBgEFBQcBAQRUMFIwUAYIKwYBBQUHMAKGRGh0dHBzOi8vYWxpeXVuLXRwbS1jYS5vc3MtY24tYmVpamluZy5hbGl5dW5jcy5jb20vcGtpMDAxL2VrbWYtY2EuY3J0MFUGA1UdHwROMEwwSqBIoEaGRGh0dHBzOi8vYWxpeXVuLXRwbS1jYS5vc3MtY24tYmVpamluZy5hbGl5dW5jcy5jb20vcGtpMDAxL2VrbWYtY2EuY3JsMA0GCSqGSIb3DQEBCwUAA4IBAQDQAJDuG9pxYkX3poXGbub+HkyYC2OuwYsEcRP0QJL2XcSePDjwQpz8p3Y8usCVVSke3gn5JhN2ATtQCpNsR1+RD1+9s1VlvwneP4vfF/9grDDs7TAiCQz+irMoFZS6TwW6Q9WW5TH9DfjUAsyPb0z4ysF4fqby+2zUM5kjPvGoi+vbXbTo/aweLgRC3tHMc/q2aj4unDKfIdB1wsGwWAu+m5BMO0NxziljC/9UDfGhrRUNUbfLw+KNcJwpRlphFgwMKDVPSmcXbU/laL6eikHJTW78s36gEcN3aWdBuzITrUvN1dyX3bMjjAO6dqj3K1d9UaSJpGguIyPmNh1rBhix", "aik_tpm": "ARgAAQALAAUAcgAAABAAFAALCAAAAAAAAQC7R7SiAAExqqCZJ60cTJXxcYMCRsctsh96vX/f2T31DMrB6SnCMV9euHlMUCUs2HT5mT2uvB+sBgy4pCMrWUtsldFuvfZtwu0XVsoXmnFiVEV6gYTkhC+CQwQNKNzp3m2lB2UojHGXGMscq8Ka7yiDse8tYhFshVFNMS1j2xnK4g0fdkVBv+oaArvB6A/XlVasuLZGvrQRPa/qr7Wqvc6qk2eSm74NLIqRf7PdzGtuYsqWBhWc4wpiEKvJn7vvcXJZLz6X7buWfBTpV6/KfDTjK7QnFOkwXw/4Y8QXAriegXAbt2bcF0tmnFa6XKuGCg2zW3W7ixNlrG9EpT1SpxCz", "ip": "127.0.0.1","port": 9002}' | jq . +keylime_create_policy -m /sys/kernel/security/ima/ascii_runtime_measurements -o runtime_policy.json +cat runtime_policy.json | jq . ``` -此时再查看发现新增对应的agent了, 使用该命令 +使用keylime_tenant来添加runtime_policy： ```shell -# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8891/v2.1/agents" | jq . +keylime_tenant -c update --uuid d432fbb3-d2f1-4a97-9ef7-75bd81c00000 \ + -t 120.26.100.138 -v 121.43.60.253 --runtime-policy /root/runtime_policy.json \ + --runtime-policy-name=tpm --cert /var/lib/keylime/cv_ca ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691742604676-6f20a3d9-1380-4465-9604-c701a501944d.png) - -### verifier - -#### GET /v2.1/agents/{agent_id:UUID} - -从CV中获取agent `agent_id`的状态。 - -具体命令如下： +查看runtime policy ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8881/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://127.0.0.1:8881/v2.1/allowlists/tpm" | jq . +{ + "code": 200, + "status": "Success", + "results": { + "name": "tpm", + "tpm_policy": null, + "runtime_policy": ... +} ``` -测试截图 - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691746520169-0152c2ab-32b4-477f-a196-88318925e1a2.png) - -#### PUT /v2.1/agents/{agent_id:UUID}/stop - -停止对 `agent_id` 的 cv 轮询，但不要删除（对于已经启动的 agent_id）。 +#### 监控IMA错误 -具体命令如下： +从verifier的日志可以看到有一些没有进行IMA签名，无法验证，所有直接报错 ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k -X PUT "https://127.0.0.1:8881/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000/stop" | jq . +2023-09-07 15:20:10.295 - keylime.tpm - INFO - Checking IMA measurement list on agent: + d432fbb3-d2f1-4a97-9ef7-75bd81c00000 +2023-09-07 15:20:10.295 - keylime.ima - WARNING - Hashes for file boot_aggregate don't match + 1aa841ace294d93414158a2f070c92d078c464da0110269d3ad1e59367cdc285 not in + ['fd2cf72bae331c6ba3db242e04f65ad5ca5c9da3f94dd5c78f5e56496e7cf0da'] +2023-09-07 15:20:10.296 - keylime.ima - ERROR - IMA ERRORS: Some entries couldn't be validated. Number of + failures in modes: ImaSig 1. +2023-09-07 15:20:10.357 - keylime.verifier - WARNING - Agent d432fbb3-d2f1-4a97-9ef7-75bd81c00000 failed, + stopping polling ``` -测试截图 - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691746611789-3697624c-7bfc-44ef-ba22-c4b4f7144a7c.png) - -#### DELETE /v2.1/agents/{agent_id:UUID} - -删除 agent_id实例。 +## 用Restful API去监控/管理Anolis OS上的各个keylime组件 +### registrar -具体命令如下： +使用registrar的RESTful APIs能够对agent进行注册、查询、删除、激活等操作。注意以下示例中的registrar IP需要根据实际IP进行修改。 -```shell -# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k -X DELETE "https://127.0.0.1:8881/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" | jq . -``` +#### GET /v2.1/agents/ -然后执行以下命令查看: +用来获取注册的agents列表，具体命令如下： ```shell -# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://121.0.0.1:8881/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://127.0.0.1:8891/v2.1/agents" | jq . +{ + "code": 200, + "status": "Success", + "results": { + "uuids": [ + "d432fbb3-d2f1-4a97-9ef7-75bd81c00000" + ] + } +} ``` -测试截图 - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691746838009-ee7d521a-de01-469a-badd-df3a56d46cc4.png) -#### GET /v2.1/allowlists/{runtime_policy_name:string} - -从 CV 中检索命名的运行时策略 runtime_policy_name。 +#### GET /v2.1/agents/{agent_id:UUID} -具体命令如下： +获取对应agent的端口、IP、EK证书等信息, 命令如下： ```shell -# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8881/v2.1/allowlists/tpm" | jq . -# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8881/v2.1/allowlists/test" | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://127.0.0.1:8891/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" | jq . +{ + "code": 200, + "status": "Success", + "results": { + ... + "ip": "121.43.60.253", + "port": 9002, + "regcount": 1 + } +} ``` -tpm的policy创建了，所以是有的 - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691753046322-299ecabe-cea7-474f-8c3c-16b21cf3604d.png) - -因为test我们没有创建和添加，所以查询是没有的 - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691753198206-4ba199ee-0943-4b10-aeaa-f656876c2cac.png) - -#### DELETE /v2.1/allowlist/{runtime_policy_name:string} - -删除 IMA policy `runtime_policy_name`. +#### PUT /v2.1/agents/{agent_id:UUID}/activate -删除已有的`tpm` policy，然后再测试，发现`tpm`没了 +激活agent_id的agent，注意**这是一个http请求，不是https，如果用https会提示这个不是TLS接口**。命令如下: ```shell -# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8881/v2.1/allowlists/tpm" | jq . -# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -X DELETE -k "https://127.0.0.1:8881/v2.1/allowlists/tpm" | jq . -# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8881/v2.1/allowlists/test" | jq . +# curl -k \ +-X PUT "http://127.0.0.1:8890/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000/activate" \ + -H 'Content-Type: application/json' \ + -d '{"auth_tag":"166be150040c57b4e2c69ad7a5dd4c57059e5838a1df4715872f6e385e8ce1ed91"}' \ + | jq . +{ + "code": 200, + "status": "Success", + "results": {} +} ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691753549321-03ad1af9-5dd3-4fb9-88d6-7cb1f5bd78f8.png) +#### DELETE /v2.1/agents/{agent_id:UUID} -删除一个不存在的policy test，会报错 +从registrar中移除ID为agent_id的agent，移除后再查看发现没有该agent了, 使用该命令 ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -X DELETE -k "https://127.0.0.1:8881/v2.1/allowlists/test" | jq . +# curl -k \ +-X PUT "http://127.0.0.1:8890/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000/activate" \ + -H 'Content-Type: application/json' \ + -d '{"auth_tag":"166be150040c57b4e2c69ad7a5dd4c57059e5838a1df4715872f6e385e8ce1ed91"}' \ + | jq . +{ + "code": 200, + "status": "Success", + "results": {} +} +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://127.0.0.1:8891/v2.1/agents" | jq . +{ + "code": 200, + "status": "Success", + "results": { + "uuids": [] + } +} ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691753558088-3010c876-d199-4432-93ba-129d37242dc3.png) - -### agent - -#### GET /version +#### POST /v2.1/agents/{agent_id:UUID} -Returns what API version the agent supports. This endpoint might not be implemented by all agents. +注册agent_id的agent到registrar.**这是一个http不是https的请求。** 注册及注册后的查询命令如下 -对应的命令如下: ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:9002/version" | jq . +# curl -X POST "http://127.0.0.1:8890/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" + -H 'Content-Type: application/json' \ + -d '{ "ekcert": "MIIE3DCCA8SgAwIBAgIBBDANBgkqhkiG9w0BAQsFADBvMQswCQYDVQQGEwJDTjEPMA0GA1UECgwGQWxpeXVuMTIwMA" \ + "aik_tpm": "ARgAAQALAAUAcgAAABAAFAALCAAAAAAAAQC7R7SiAAExqqCZJ60cTJXxcYMCRsctsh96vX/f2T31DMrB6SnCMV9euHlMUCUs" \ +"ip": "127.0.0.1","port": 9002}' | jq . +{ + "code": 200, + "status": "Success", + ... +} +# curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://127.0.0.1:8891/v2.1/agents" | jq . +{ + "code": 200, + "status": "Success", + "results": { + "uuids": [ + "d432fbb3-d2f1-4a97-9ef7-75bd81c00000" + ] + } +} ``` -结果截图如下： - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691745013033-9ff01fcc-b757-4b2a-93c4-a1353694aa68.png) - -#### GET /v2.1/keys/pubkey +### verifier -获取agent的公钥. +#### GET /v2.1/agents/{agent_id:UUID} -对应的命令如下: +从CV中获取agent `agent_id`的状态。具体命令如下： ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:9002/v2.1/keys/pubkey" | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://127.0.0.1:8881/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" | jq . +{ + "code": 200, + "status": "Success", + "results": { + ... + "hash_alg": "sha256", + "enc_alg": "rsa", + "sign_alg": "rsassa", + "verifier_id": "default", + "verifier_ip": "121.43.60.253", + "verifier_port": 8881, + "severity_level": 6, + } +} ``` -结果截图如下： - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691745101144-e91da717-43e8-4592-b2fb-e2b2f67ebe12.png) - -#### GET /v2.1/quotes/identity - -Get identity quote from node +#### PUT /v2.1/agents/{agent_id:UUID}/stop -对应的命令如下: +停止对 `agent_id` 的 cv 轮询，但不要删除（对于已经启动的 agent_id）。具体命令如下： ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:9002/v2.1/quotes/identity?nonce=1234567890ABCDEFHIJ" | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -k \ + -X PUT "https://127.0.0.1:8881/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000/stop" \ + | jq . +{ + "code": 200, + "status": "Success", + "results": {} +} ``` -结果截图如下： - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1690945718718-2333ced3-34bf-47a4-824b-fcac9805bfaf.png) - -#### GET /v2.1/quotes/integrity +#### DELETE /v2.1/agents/{agent_id:UUID} -Get integrity quote from node +删除 agent_id实例。删除包括删除后的查看命令如下： ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:9002/v2.1/quotes/integrity?nonce=1234567890ABCDEFHIJ&mask=0x10401&partial=0" | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k -X DELETE "https://127.0.0.1:8881/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" \ + | jq . +{ + "code": 200, + "status": "Success", + "results": {} +} +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://121.0.0.1:8881/v2.1/agents/d432fbb3-d2f1-4a97-9ef7-75bd81c00000" | jq . +{ + "code": 404, + "status": "agent id not found", + "results": {} +} ``` -结果截图如下： - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691745220787-b9613f49-efd2-4bd9-a9a2-a4a5a734f35d.png) - -#### GET /v2.1/keys/verify - -Get confirmation of bootstrap key derivation +#### GET /v2.1/allowlists/{runtime_policy_name:string} -对应的命令如下： +从 CV 中检索命名的运行时策略 runtime_policy_name。比如tpm的policy创建了，可以通过以下命令查看 ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:9002/v2.1/keys/verify?challenge=1234567890ABCDEFHIJ" | jq . +curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -k \ + "https://127.0.0.1:8881/v2.1/allowlists/tpm" | jq . +{ + "code": 200, + "status": "Success", + "results": { + "name": "tpm", + "tpm_policy": null, + "runtime_policy": ... + } +} ``` -截图结果如下(TPM实例没有bootstrap key）： - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691745331527-7a3e22be-dcd3-477f-ab8c-696f2560c39e.png) - -## Anolis OS上keylime高级功能实践 -### User Selected PCR Monitoring - -参考: -- [User Selected PCR Monitoring](https://keylime.readthedocs.io/en/latest/user_guide/user_selected_pcr_monitoring.html) - -缺点： -- PCR数量优先，扩展性不好 - -配置tpm_policy并用keylime_tenant工具进行添加，具体命令如下 +对于没有创建的test policy，查询是没有的 ```shell -# keylime_tenant -v 121.43.60.253 -t 120.26.100.138 --uuid d432fbb3-d2f1-4a97-9ef7-75bd81c00000 --tpm_policy "{\"22\": [\"0000000000000000000000000000000000000001\", \"0000000000000000000000000000000000000000000000000000000000000001\", \"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001\", \"ffffffffffffffffffffffffffffffffffffffff\", \"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\", \"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff\"], \"15\": [\"0000000000000000000000000000000000000000\", \"0000000000000000000000000000000000000000000000000000000000000000\", \"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000\"]}" -c add --cert /var/lib/keylime/cv_ca -INFO:keylime.config:Reading configuration from ['/etc/keylime/logging.conf'] +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -k \ + "https://127.0.0.1:8881/v2.1/allowlists/test" | jq . +{ + "code": 404, + "status": "Runtime policy test not found", + "results": {} +} ``` -对应的截图如下： - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691746192516-c5d84e2e-c677-438d-9741-c3d63b2ef7b5.png) - -#### 监控 - -成功的case如下（agent时刻监控TPM PCRs的状态） - -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691746272947-2c5dc3b8-bf2c-4bfc-8c2d-2b14e6af0dbe.png) - -### Use Measured Boot - -参考： -- [Use Measured Boot](https://keylime.readthedocs.io/en/latest/user_guide/use_measured_boot.html) +#### DELETE /v2.1/allowlist/{runtime_policy_name:string} -第一步，生成measure boot policy +删除 IMA policy `runtime_policy_name`. 比如删除已有的`tpm` policy，然后再测试，发现该policy没了 ```shell -cd keylime -./scripts/create_mb_refstate -i /sys/kernel/security/tpm0/binary_bios_measurements ./measured_boot_reference_state.json -cat ./measured_boot_reference_state.json | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://127.0.0.1:8881/v2.1/allowlists/tpm" | jq . +{ + "code": 200, + "status": "Success", + "results": { + "name": "tpm", + "tpm_policy": null, + "runtime_policy": ... + } +} +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -X DELETE \ + -k "https://127.0.0.1:8881/v2.1/allowlists/tpm" | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt \ + -k "https://127.0.0.1:8881/v2.1/allowlists/test" | jq . +{ + "code": 404, + "status": "Runtime policy test not found", + "results": {} +} ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691751357999-b9d6393d-3c1e-443c-b504-1a457ea7b5e0.png) - -用keylime_tenant工具进行添加mb_reference，对应的命令如下： +而删除一个不存在的policy test，会报错 ```shell -keylime_tenant -c update -t 120.26.100.138 -v 121.43.60.253 -u d432fbb3-d2f1-4a97-9ef7-75bd81c00000 --mb_refstate ./measured_boot_reference_state.json --cert /var/lib/keylime/cv_ca +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -X DELETE \ + -k "https://127.0.0.1:8881/v2.1/allowlists/test" | jq . +{ + "code": 404, + "status": "Runtime policy test not found", + "results": {} +} ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691751495723-a3ed228e-6521-4e46-b551-8d6855d2042e.png) +### agent -agent侧查看轮询结果:时刻轮询/监控着是否有异常。 +#### GET /version -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691751506214-660ed997-28b1-4662-afcd-6a31bd78802e.png) +获取agent支持的API版本。对应的命令如下: -### Runtime Integrity Monitoring +```shell +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -k \ + "https://127.0.0.1:9002/version" | jq . +{ + "code": 200, + "status": "Success", + "results": { + "supported_version": "2.1" + } +} +``` -参考: -- [Runtime Integrity Monitoring](https://keylime.readthedocs.io/en/latest/user_guide/use_measured_boot.html) +#### GET /v2.1/keys/pubkey -使用keylime_create_policy工具来生成policy： +获取agent的公钥. 对应的命令如下: ```shell -keylime_create_policy -m /sys/kernel/security/ima/ascii_runtime_measurements -o runtime_policy.json -cat runtime_policy.json | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -k \ + "https://127.0.0.1:9002/v2.1/keys/pubkey" | jq . +{ + "code": 200, + "status": "Success", + "results": { + "pubkey": ... + } +} ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691752688123-c8058faf-b4f5-404e-aa08-b130d3f8e8c1.png) +#### GET /v2.1/quotes/identity -使用keylime_tenant来添加runtime_policy： +从节点获取identity quote, 对应的命令如下: ```shell -keylime_tenant -c update --uuid d432fbb3-d2f1-4a97-9ef7-75bd81c00000 -t 120.26.100.138 -v 121.43.60.253 --runtime-policy /root/runtime_policy.json --runtime-policy-name=tpm --cert /var/lib/keylime/cv_ca +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -k \ + "https://127.0.0.1:9002/v2.1/quotes/identity?nonce=1234567890ABCDEFHIJ" \ + | jq . +{ + "code": 200, + "status": "Success", + "results": { + "quote": ... + "hash_alg": "sha256", + "enc_alg": "rsa", + "sign_alg": "rsassa", + "pubkey": ... + } +} ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691752701674-c4b65354-2bb5-4d90-8653-b854569ec572.png) +#### GET /v2.1/quotes/integrity -查看runtime policy +从节点获取integrity quote，具体命令如下： ```shell -curl --key /var/lib/keylime/cv_ca/client-private.pem --cert /var/lib/keylime/cv_ca/client-cert.crt -k "https://127.0.0.1:8881/v2.1/allowlists/tpm" | jq . +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -k \ +"https://127.0.0.1:9002/v2.1/quotes/integrity?nonce=1234567890&mask=0x10401&partial=0" +{ + "code": 200, + "status": "Success", + "results": { + "quote": ... + "hash_alg": "sha256", + "enc_alg": "rsa", + "sign_alg": "rsassa", + "pubkey": ... + "ima_measurement_list": ... + "mb_measurement_list": ... + "ima_measurement_list_entry": 0 + } +} ``` -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691752726734-65857254-a3fb-4316-ba32-dab701da582e.png) - -#### 监控IMA错误 +#### GET /v2.1/keys/verify -从verifier的日志可以看到有一些没有进行IMA签名，无法验证，所有直接报错 +获取bootstrap key的验证, 对应的命令如下： -![undefined](https://intranetproxy.alipay.com/skylark/lark/0/2023/png/136439/1691752719183-3f8eb685-0e20-4f8d-b361-0d9398eec49e.png) +```shell +# curl --key /var/lib/keylime/cv_ca/client-private.pem \ + --cert /var/lib/keylime/cv_ca/client-cert.crt -k \ + "https://127.0.0.1:9002/v2.1/keys/verify?challenge=1234567890ABCDEFHIJ" | jq . +{ + "code": 400, + "status": "Bootstrap key not yet available.", + "results": {} +} +``` -- Gitee