//Cloud notes from my desk -Maheshk

"Fortunate are those who take the first steps.” ― Paulo Coelho

[Linux] Commands to collect CPU, BIOS, MEM, HW details in one go

At times, you wanted to run a script in on-prem or in cloud to understand the Linux Machine spec, this script (series of commands) helps you to get that info easily.

1. To get the CPU info,
$ lscpu
2. To get the H/w info,
$ lshw -short
3. To get the H/w info2,
$ hwinfo –short
4. To get the Diskblocks,
$ lsblk
5. To get the mounted disk,
$ df -hT
6. To get the attached disk,
$ fdisk -l
7. To get the memory,
$ free -m
8. To get the OSVersion,
$ cat /proc/version
9. To get the HW CPU,
$ dmidecode -t processor (cpu), $ dmidecode -t memory (memory), $ dmidecode -t bios (bios)

Hint: You could save this as bash script(#/bin/bash) and then cat out to individual file to /tmp/ so that we can get all of them in one go.

Advertisements

2019-05-07 Posted by | Linux, Uncategorized | | Leave a comment

Failed to delete virtual network ‘k8s-vkdemo-2-vnet’. Error: Subnet virtual-node-aci is in use by and cannot be deleted

Recently I tried deleting few resource groups(RG) having AKS cluster with ACI enabled. I couldn’t get them deleted completely for some reason leaving Vnet all those RG’s.

Error message was “Failed to delete virtual network ‘k8s-vkdemo-2-vnet’. Error: Subnet virtual-node-aci is in use by and cannot be deleted.” I got the same error for other 2 cluster RG as well. I tried deleting individual sub-nets there I got an error.

With help of support and this article I was able to delete the RG. It’s is documented here -> https://docs.microsoft.com/en-us/azure/container-instances/container-instances-vnet#delete-network-resources.

Here is the revised script.

# Replace <my-resource-group> with the name of your resource group
RES_GROUP=<my-resource-group>

# Replace <my_vnet_name> with the name of your VNet name
VNET_NAME=<my_vnet_name>

# Replace <my_subnet_name> with the name of your subnet name
SUBNET_NAME=<my_subnet_name>

# Get network profile ID
NETWORK_PROFILE_ID=$(az network profile list --resource-group $RES_GROUP --query [0].id --output tsv)

# Delete the network profile
az network profile delete --id $NETWORK_PROFILE_ID -y

# Get the service association link (SAL) ID
SAL_ID=$(az network vnet subnet show --resource-group $RES_GROUP --vnet-name $VNET_NAME --name $SUBNET_NAME --query id --output tsv)/providers/Microsoft.ContainerInstance/serviceAssociationLinks/default

# Delete the default SAL ID for the subnet
az resource delete --ids $SAL_ID --api-version 2018-07-01

# Delete the subnet delegation to Azure Container Instances
az network vnet subnet update --resource-group $RES_GROUP --vnet-name $VNET_NAME --name $SUBNET_NAME--remove delegations 0

# Delete the subnet
az network vnet subnet delete --resource-group $RES_GROUP --vnet-name $VNET_NAME --name $SUBNET_NAME

# Delete virtual network
az network vnet delete --resource-group $RES_GROUP --name $VNET_NAME

2019-03-18 Posted by | AKS, Azure, Azure Dev, Uncategorized | Leave a comment

Why Azure Kubernetes Service(AKS) vs Others

What is AKS?
– deploy a managed Kubernetes cluster in Azure.
– reduces the complexity and operation overhead of managing
K8s by offloading much of that responsibility to Azure
– handles critical tasks like health monitoring and maintenance for you.
– masters are managed by Azure and You only manage and maintain the agent nodes.
– free, you only pay for the agent nodes and not for the master

tst3

Why AKS vs Others?
– Streamlined application onboarding with integrated VSTS CI/CD via DevOps Project
– Deep integration with Azure Monitor and Log Search
– Using Azure Dev Spaces for AKS – enables multiple developers to collaborate and rapidly iterate/debug microservices directly in AKS dev environment
– Open source thought leadership through projects like Virtual Kubelet, Helm, Draft, Brigade & Kashti & our contribution to the open source community
– Support for scenarios such as elastic bursting using Azure Container Instance (ACI) and Virtual Kubelet
– Users can use Key Vault for increased security and control over Kubernetes keys and passwords, create and import encryption keys in minutes
– Developers and operations can be assured their workloads will have Automated OS & Framework Patching with ACR Build
– Rich Tooling Support  VS Code/VS integration (VSCode is a free code editor; try today, you’ll thank us )

Best practice guidance
———————-
> For integration with existing virtual networks or on-premises networks, use advanced networking in AKS.
> greater separation of resources and controls in an enterprise environment

Two different ways to deploy AKS clusters into virtual networks:
+ Basic networking – Azure manages the virtual network resources as the cluster is deployed and uses the kubenet Kubernetes plugin.
+ Advanced networking – Deploys into an existing virtual network, and uses the Azure Container Networking Interface (CNI) Kubernetes plugin. Pods receive individual IPs that can route to other network services or on-premises resources.
The Container Networking Interface (CNI) is a vendor-neutral protocol that lets the container runtime make requests to a network provider. The Azure CNI assigns IP addresses to pods and nodes, and provides IP address management (IPAM) features as you connect to existing Azure virtual networks. Each node and pod resource receives an IP address in the Azure virtual network, and no additional routing is needed to communicate

$ az aks create –resource-group myAKSCluster –name myAKSCluster –generate-ssh-keys \
–aad-server-app-id \
–aad-server-app-secret \
–aad-client-app-id \
–aad-tenant-id

$ az aks get-credentials –resource-group myAKSCluster –name myAKSCluster –admin
Merged “myCluster” as current context ..

$ kubectl get nodes

NAME STATUS ROLES AGE VERSION
aks-nodepool1-42032720-0 Ready agent 1h v1.9.6
aks-nodepool1-42032720-1 Ready agent 1h v1.9.6
aks-nodepool1-42032720-2 Ready agent 1h v1.9.6

2019-03-06 Posted by | AKS, Azure Dev, Kubernetes, Linux, Microservices, PaaS | | Leave a comment

[k8s] Kubernetes dashboard access warnings

Accessing your Kubernetes dashboard through proxy you might experience this warning.

Sample text: configmaps is forbidden: User “system:serviceaccount:kube-system:kubernetes-dashboard” cannot list configmaps in the namespace “default” k8rbac

Resolution: From the message it is apparent that, access to the dashboard is restricted. Solution is to add the required rolebinding as below.

Two ways to do it. You can create the binding with simple one liner from CLI or YAML way.

$ kubectl create clusterrolebinding kubernetes-dashboard --clusterrole=cluster-admin --serviceaccount=kube-system:kubernetes-dashboard

or  YAML way to create the role binding. Create the below Yaml file with some name say “dashboard-rolebinding.yaml” and submit for creation in the same Kubectl.

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
labels:
k8s-app: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system

$ kubectl create -f dashboard-rolebinding.yaml

PS- I had this experience when I access my AKS cluster, so not sure about other providers or distribution at this time of writing.

update:

$kubectl proxy

http://localhost:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/#!/node?namespace=default

 

2019-01-26 Posted by | Kubernetes, Uncategorized | | Leave a comment

[LFCS] Managing Software RAID

mdadm is a super cool command in Linux used to manage MD devices aka Linux Software RAID. Before we jump in, let’s see what is RAID. – –Redunant array of independent disks
–if disk gets corrupted, then data loss
–using RAID, if one disk fails, other will take over

$ man page says this,

RAID devices are virtual devices created from two or more real block devices. This allows multiple devices (typically disk drives or partitions thereof) to be combined into a single device to hold (for example) a single filesystem. Some RAID levels include redundancy and so can survive some degree of device failure.

Understanding RAID soln,
RAID O- Striping { one big device based on multiple disk, no redundancy or easy recovery}
RAID 1- Mirroring { 2 disks, identical }
RAID 5- striping with distributed parity { if data is written with parity info, if one disks fails, then restore the data }
RAID 6- striping with dual distributed parity { redundant parity is written,advancement of RAID 5 }
RAID 10- mirrored and striped { minimum of 10 disks, 2 for striping, 2 for mirrored}

Sample question: How to create a RAID 5 Device using 3 disk device of 1 GB each. Also allocate additional device as spare device.
— Put a file system on it and mount it on /raid
— Fail one of the devices, monitor what is happening
— Replace the failed device with spare device

Solution:

$ cat /proc/partitions
$ fdisk -l { list the partition tables for the device, if no device specified then list all the partitions from the system }
$ fdisk /dev/sdc
-n create a new partitions { size as +1G }
-m {help}
-t :L {enter “fd” for Linux raid auto}
-w { write the
entries to persist }
$ partprobe { inform the OS partition table changes }
$ vim /etc/fstab { before we proceed, let’s verify the disks are not used for any mounting. In my case, I had used as swap device mounting so got an error saying device is busy error. Rmv the entry, reboot }
$ mdadm –create /dev/md1 -l 5 -x 1 –raid-disk=3 /dev/sdc1 /dev/sdc2 /dev/sdc3 /dev/sdc4 –verbose –auto=yes
$ mdadm –detail /dev/md1 { list details after creation, should see 3 device + 1 spare device }

$ mdadm fail dev/md1 /dev/sdc1 { to simulate the failure }
$ mdadm –remove /dev/md1 /dev/sdc1 { remove the faulty one }
$ mdadm –add /dev/md1 /dev/sdc1 { add the device back to the pool as spare device if healthy }

other disk related commands,
$ blkid $ blkid /dev/sdc
$ df -h, df -h -T, df -hT /home
$ du -h /home, du -sh /home/mydir
$ mount /dev/sdc5 /mnt, cd /mnt , touch file1 { after mounting make entry in /etc/fstab to persist}
$ mount -a { to mount all fs mentioned in fstab}
$ mkfs.ext4 /dev/sda4 { format a partition of type ext4, after creating a partition }

Command output:

root@mikky100:~# mdadm –fail /dev/md1 /dev/sdc1 { Simulate the failure }
mdadm: set /dev/sdc1 faulty in /dev/md1

root@mikky100:~# mdadm –detail /dev/md1 { view the detail after the failure, we should see the spare disk getting rebuild }
/dev/md1:
Version : 1.2
Creation Time : Mon Jun 11 06:10:34 2018
Raid Level : raid5
Array Size : 1951744 (1906.32 MiB 1998.59 MB)
Used Dev Size : 975872 (953.16 MiB 999.29 MB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jun 11 17:06:09 2018
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 1
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 512K

Rebuild Status : 3% complete

Name : mikky100:1 (local to host mikky100)
UUID : 772f743c:b1209727:6910411d:690d6294
Events : 20

Number Major Minor RaidDevice State
3 8 36 0 spare rebuilding /dev/sdc4
1 8 34 1 active sync /dev/sdc2
4 8 35 2 active sync /dev/sdc3

0 8 33 – faulty /dev/sdc1

root@mikky100:~# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Mon Jun 11 06:10:34 2018
Raid Level : raid5
Array Size : 1951744 (1906.32 MiB 1998.59 MB)
Used Dev Size : 975872 (953.16 MiB 999.29 MB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jun 11 17:08:13 2018
State : clean
Active Devices : 3
Working Devices : 3
Failed Devices : 1
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : mikky100:1 (local to host mikky100)
UUID : 772f743c:b1209727:6910411d:690d6294
Events : 37

Number Major Minor RaidDevice State
3 8 36 0 active sync /dev/sdc4
1 8 34 1 active sync /dev/sdc2
4 8 35 2 active sync /dev/sdc3

0 8 33 – faulty /dev/sdc1

root@mikky100:~# mdadm –add /dev/md1 /dev/sdc1 { add the disk back as spare }

root@mikky100:~# mdadm –detail /dev/md1
/dev/md1:
Version : 1.2
Creation Time : Mon Jun 11 06:10:34 2018
Raid Level : raid5
Array Size : 1951744 (1906.32 MiB 1998.59 MB)
Used Dev Size : 975872 (953.16 MiB 999.29 MB)
Raid Devices : 3
Total Devices : 4
Persistence : Superblock is persistent

Update Time : Mon Jun 11 17:12:21 2018
State : clean
Active Devices : 3
Working Devices : 4
Failed Devices : 0
Spare Devices : 1

Layout : left-symmetric
Chunk Size : 512K

Name : mikky100:1 (local to host mikky100)
UUID : 772f743c:b1209727:6910411d:690d6294
Events : 39

Number Major Minor RaidDevice State
3 8 36 0 active sync /dev/sdc4
1 8 34 1 active sync /dev/sdc2
4 8 35 2 active sync /dev/sdc3

5 8 33 – spare /dev/sdc1

2018-06-11 Posted by | LFCS, Linux, OSS | | Leave a comment

AI Revolution and resources

Before 1990, its was all CLI. Every software had commands & parameters. That was the standard of the Software development. Post that we had GUI based interfaces having buttons and mouse clicks. Around 1995, we had Internet Web UI as became the standard. After the success of smart phones in 2008, we saw more Responsive UI developed using jQuery,Bootstrap and now we live in the era of Conversation UI. Bots like Cortana, Siri, Alexa are around the block to help our modern needs.

Below is the snippet which I borrowed from this ebook covering “how the enterprise information technology has transformed over the last few decades”. Thanks to the author who put the classification beautifully with examples.

Client-Server Revolution → Systems of records It was the client-server revolution that first enabled broad use of information technology to manage business. Organizations first built systems of records: Customer Relationship Management (CRM) systems; Human Capital Management (HCM) systems for HR; and Enterprise Resource Planning (ERP) systems for financials and key assets.

Internet Revolution → System of engagement The rise of the internet, mobile, and chat allowed us to create systems of engagement that interfaced between the systems of records and interacted directly with customers and suppliers.

AI Revolution → System of intelligence What is emerging now are systems of intelligence that integrate data across all systems of record, connect directly to systems of engagement, and build systems that understand and reason with the data. These systems can drive workflows and management processes, optimize operations, and drive intelligent interactions with customers, employees, suppliers, and stakeholders.

Below is our Microsoft AI platform story covered in one slide deck.

image

If you are wondering how your organization can start the AI journey today, then below is the some of the key resources for learning,

  1. Azure AI -> A page where our Microsoft AI story is well articulated (In the page, scroll down for ‘AI Services’).
  2. Intelligent KIOSK –> must try windows app to demonstrate Pre-built AI (Cognitive APIs) & store url
  3. Seeing AI –> Great demo app on iOS using our cognitive APIs. Good to install and check the capability it brings in
  4. Conference Buddy –> ingredients needed to develop an intelligent chatbot. sample code for try one.
  5. Microsoft AI School –> Another great learning resource our Services and ML offerings
  6. The JFK Files –> Cognitive Search – An AI-first approach to content understanding – code

update: 6/26
http://unearth.ai/
https://alice.unearth.ai/
http://wolterskluwereap.azurewebsites.net/Home/Search
https://text-analytics-demo-dev.azurewebsites.net/

update 12/28

https://docs.microsoft.com/en-us/azure/cognitive-services/Text-Analytics/Tutorials/tutorial-power-bi-key-phrases
https://rare-technologies.com/text-summarization-with-gensim/
http://webmedsearch.azurewebsites.net/
https://github.com/liamca
https://github.com/liamca/medical-ner-search
https://github.com/liamca/BM25_Key_Phrase_Extraction
https://github.com/liamca/keyphrase_extraction_and_summarization_over_custom_content

Happy Learning !!

2018-06-02 Posted by | AI, Azure, ML | | Leave a comment

%d bloggers like this: