Questions to use Azure CNI and Kubernetes Baremetall on Azure in VMs in a non AKS Cluster Enviroment

Question

hello togehter,

i found this cni and plugin and i am currently testing it in a kubernetes v1.20.5 bare metall installation on azure with docker.
we do not use aks azure service because we want to learn and deploy later kubernetes on a prem vmware envirpment at one of our customers, but will we use there a different cni like flannel or weave.
flannel and weave does not work on azure, that is the reason why i use the azure cni plugins.

my test enviroment:

• we have 6 VMs with rhel7.9... 3 kubernets master and 3 kubenetes workers.
• we have one vnet with two subnet (192.168.x.x for nodes && 10.1.x.x. for pods)
• every VM has two nics ... eth0 node subnet 192.x AND eth1 is the pod subnet 10.x
• on all 6 VMs i installed the azure CNI with (# ./install-cni-plugin.sh v1.2.9 v0.9.1)
• on all 6 VMs i installed the azure CLI and logged in with my account
• we enabled port forwading and iptables rule like describend in your documentation
• net.ipv4.ip_forward=1
• iptables -t nat -A POSTROUTING -m addrtype ! --dst-type local ! -d 10.1.0.0/16 -j MASQUERADE

my ipam configuration below are not working right.
i need to add a second nic with

{
   "cniVersion":"0.3.0",
   "name":"azure",
   "plugins":[
      {
         "type":"azure-vnet",
         "mode":"transparent",
         "ipsToRouteViaHost":["169.254.20.10"],
         "ipam": {
           "type": "host-local",
           "subnet": "10.1.0.0/16",
           "gateway": "10.1.0.1"
         },
         "dns": {
           "nameservers": [ "10.1.0.1" ]
         }
      },
      {
         "type":"portmap",
         "capabilities":{
            "portMappings":true
         },
         "snat":true
      }
   ]
}

tail -f /var/log/azure-vnet.log

2021/03/30 12:23:11 [41456] [cni] reboot time 2021-03-29 23:16:00 +0000 UTC
2021/03/30 12:23:11 [41456] Connected to telemetry service
2021/03/30 12:23:11 [41456] [cni-net] Plugin azure-vnet version v1.2.8.
2021/03/30 12:23:11 [41456] [cni-net] Running on Linux version 4.18.0-147.43.1.el8_1.x86_64 (******@x86-vm-08.build.eng.bos.redhat.com) (gcc version 8.3.1 20190507 (Red Hat 8.3.1-4) (GCC)) #1 SMP Thu Feb 18 08:33:46 EST 2021
2021/03/30 12:23:11 [41456] [Azure-Utils] iptables --version
2021/03/30 12:23:11 [41456] [cni-net] iptable version:iptables v1.8.2 (nf_tables), err:<nil>
2021/03/30 12:23:11 [41456] [Azure-Utils] ebtables --version
2021/03/30 12:23:11 [41456] [cni-net] ebtable version ebtables 1.8.2 (nf_tables), err:<nil>
2021/03/30 12:23:11 [41456] [net] Network interface: {Index:1 MTU:65536 Name:lo HardwareAddr: Flags:up|loopback} with IP: [127.0.0.1/8]
2021/03/30 12:23:11 [41456] [net] Network interface: {Index:2 MTU:1500 Name:eth0 HardwareAddr:00:0d:3a:6c:c0:0c Flags:up|broadcast|I would be very grateful if someone would help me, as I'm slowly going insanemulticast} with IP: [192.168.0.7/24]
2021/03/30 12:23:11 [41456] [net] Network interface: {Index:3 MTU:1500 Name:eth1 HardwareAddr:00:0d:3a:f5:cb:22 Flags:up|broadcast|multicast} with IP: [10.1.1.10/16]
2021/03/30 12:23:11 [41456] [net] Network interface: {Index:4 MTU:1500 Name:docker0 HardwareAddr:02:42:0f:16:a8:5e Flags:up|broadcast|multicast} with IP: [172.17.0.1/16]
2021/03/30 12:23:11 [41456] [net] network store key not found
2021/03/30 12:23:11 [41456] [cni-net] Plugin started.
2021/03/30 12:23:11 [41456] CNI_COMMAND environment variable set to VERSION
2021/03/30 12:23:11 [41456] [cni-net] Plugin stopped

tail -f /var/log/azure-vnet-ipam.log

2021/03/29 22:37:18 [10282] [Utils] Initializing HTTP client with connection timeout: 10, response header timeout: 10
2021/03/29 22:37:18 [10282] [ipam] Wireserver call http://168.63.129.16/machine/plugins?comp=nmagent&type=getinterfaceinfov1 to retrieve IP List
2021/03/29 22:37:18 [10282] [ipam] got 0 addresses from interface eth0, subnet 192.168.0.0/24
2021/03/29 22:37:18 [10282] [ipam] got 0 addresses from interface eth1, subnet 10.1.0.0/16
2021/03/29 22:37:18 [10282] [ipam] merging address space
2021/03/29 22:37:18 [10282] [ipam] saving ipam state.
2021/03/29 22:37:18 [10282] [ipam] Save succeeded.
2021/03/29 22:37:18 [10282] [azure-vnet-ipam] Failed to release address: Pool id  not found :Invalid address pool.
2021/03/29 22:37:18 [10282] [cni-ipam] DEL command completed with err:Failed to release address: Pool id  not found :Invalid address pool.
2021/03/29 22:37:18 [10282] [cni-ipam] Plugin stopped.

Here are my questions:

is my config right at the top? or do i understand something wrong?
this works only if i add multiple ip adresse on azure to the second nic (eth1, subnet 10.1.0.0/16) but i expect something different. But i dont want to do that !!! :-(
if i understand right your documentation, i would expect that the plugin creates,adds and removes ipadresses on the 2nd nic?
I would be very grateful if someone would help me, as I'm slowly going insane^^

thx in advance
jose

Answer 1

@Jose H Freidhof Thanks for raising this question! Firstly, apologies for the delay in responding here and any inconvenience this issue may have caused.

We need to analysis the issue, I would recommend you to contact support, so If you have a support plan, I request you file a support ticket, else please do let us know, we will try and help you get a one-time free technical support. In this case, could you send an email to AzCommunity[at]Microsoft[dot]com referencing this thread as well as your subscription ID. Please mention "ATTN subm" in the subject field. Thank you for your cooperation on this matter and look forward to your reply.

Thanks for your patience and co-operation.

Hope this helps!

-----------------------------------------------------------------------------------------------------------------------------

Please do not forget to "Accept the answer” and “up-vote” wherever the information provided helps you, this can be beneficial to other community members.