//Cloud notes from my desk -Maheshk

"Fortunate are those who take the first steps.” ― Paulo Coelho

[Azure Billing] Sample REST API postman request

Azure Billing REST API helps to predict and manage Azure costs.  Following API’s are there to query the billing data.

1) Resource Usage (Preview)

Get consumption data for an Azure subscription- https://msdn.microsoft.com/en-us/library/azure/mt219001.aspx

Sample working query:-
https://management.azure.com/subscriptions/your-subscription-id/providers/Microsoft.Commerce/UsageAggregates?api-version=2015-06-01-preview&reportedStartTime=2014-05-01T00%3a00%3a00%2b00%3a00&reportedEndTime=2017-04-01T00%3a00%3a00%2b00%3a00&aggregationGranularity=Daily&showDetails=false

image

2) Resource RateCard (Preview)

Get price and metadata information for resources used in an Azure subscription- https://msdn.microsoft.com/en-us/library/azure/mt219004.aspx

Sample working query:-
https://management.azure.com/subscriptions/your-subscription-id/providers/Microsoft.Commerce/RateCard?api-version=2016-08-31-preview&$filter=OfferDurableId eq ‘MS-AZR-0063P’ and Currency eq ‘USD’ and Locale eq ‘en-US’ and RegionInfo eq ‘US’

image

 

Ps:- Assuming you have a valid bearer token for authorization.

https://docs.microsoft.com/en-us/rest/api/index?redirectedfrom=MSDN#send-the-request

2017-04-28 Posted by | Azure, Billing, PaaS | | Leave a comment

[Azure Powershell] How to improve the performance of BlobCopy by placing inline C#.NET code

Performance issues are always hard and tricky to fix. Recently I had this ask to check why the given Powershell execution takes time and gets slower on heavy iteration. In the beginning, copy operation used to take only few mins but over a heavy iteration, we saw the performance degraded to 2x, 3x..

Sample looping of copy command,

    foreach($SrcBlob in $SrcBlobs)
    {
          $DestBlob = “root/” + $SrcBlob.Name
          Start-AzureStorageBlobCopy -SrcBlob $SrcBlob.Name -SrcContainer $SrcContainerName -Context $SrcContext -DestBlob $DestBlob -DestContainer $DestContainerName -DestContext $DestContext -Force
    }

We have also used measure-command to list the execution duration and print the value for each iteration and also the sum at the end of the loop to confirm. We ran the loop by copying 5000+ times b/w storage accounts and found that, the Powershell command let execution gets slower on iteration for some reason.  Later on investigation, we confirmed this was due to known limitation with Powershell.

   $CopyTime = Measure-Command{

         Start-AzureStorageBlobCopy -SrcBlob $SrcBlob.Name -SrcContainer $SrcContainerName -Context $SrcContext -DestBlob $DestBlob -DestContainer $DestContainerName -DestContext $DestContext -Force

    }

Yes, there is a known issue with PowerShell running slow while iterating large loops due to the nature how PowerShell works, as in the 16th time of the iteration the content of the loop must be compiled dynamically and the .Net will need to run some extra checks. Thanks for some of our internal folks to provide clarity on this. To overcome this, there is a workaround suggested which is what we are here. Yes, we replaced with .Net code having copy logic inside the Powershell script to improve the performance. In this way, the security check only running once instead every 16 time. You will find the detailed information here –> Why cant PowerShell run loops fast ? https://blogs.msdn.microsoft.com/anantd/2014/07/25/why-cant-powershell-run-loops-fast/

How to place the C# as inline code within Powershell:-

$reflib = (get-item “c:tempMicrosoft.WindowsAzure.Storage.dll”).fullname

[void][reflection.assembly]::LoadFrom($reflib)

$Source = @”

using Microsoft.WindowsAzure.Storage;

using Microsoft.WindowsAzure.Storage.Auth;

using Microsoft.WindowsAzure.Storage.Blob;

using System;

 

namespace ns{

public static class copyfn1{

   public static void Copy_bw_SA_Blobs_Test(string sourceAccountKey, string destAcKey, string SrcSAName, string DestSAName, string SrcContainerName, string DestContainerName, string SrcPrefix, string DestPrefix){

       StorageCredentials scSrc = new StorageCredentials(SrcSAName, sourceAccountKey);

       CloudStorageAccount srcAc = new CloudStorageAccount(scSrc, true);

       CloudBlobClient cbcSrc = srcAc.CreateCloudBlobClient();

       CloudBlobContainer contSrc = cbcSrc.GetContainerReference(SrcContainerName);

 

      //Generate SAS Key and use it for delegate access

      SharedAccessBlobPolicy sasConstraints = new SharedAccessBlobPolicy();

      sasConstraints.SharedAccessExpiryTime = DateTime.UtcNow.AddHours(24);

      sasConstraints.Permissions = SharedAccessBlobPermissions.Write

          | SharedAccessBlobPermissions.List | SharedAccessBlobPermissions.Add

          | SharedAccessBlobPermissions.Delete | SharedAccessBlobPermissions.Create

          | SharedAccessBlobPermissions.Read;

      string sasContainerToken = contSrc.GetSharedAccessSignature(sasConstraints);

      //Return the URI string for the container, including the SAS token.

      string containersas = contSrc.Uri + sasContainerToken;

      CloudBlobContainer container = new CloudBlobContainer(new Uri(containersas));

 

      //Destination account – no SAS reqd

      StorageCredentials scDst = new StorageCredentials(DestSAName, destAcKey);

      CloudStorageAccount DstAc = new CloudStorageAccount(scDst, true);

      CloudBlobClient cbcDst = DstAc.CreateCloudBlobClient();

      CloudBlobContainer contDst = cbcDst.GetContainerReference(DestContainerName);

 

      foreach (var eachblob in container.ListBlobs(SrcPrefix, true, BlobListingDetails.Copy)){

         CloudBlob srcBlob = (CloudBlob)eachblob;

         string srcpath = srcBlob.Name;

         string dstpath = (DestPrefix != “”) ? srcpath.Replace(SrcPrefix, DestPrefix) : srcpath;

                       Console.WriteLine(“Files copying-” + dstpath);

       

         if (srcBlob.BlobType == BlobType.BlockBlob){

             CloudBlockBlob dstblob = contDst.GetBlockBlobReference(dstpath);

             dstblob.StartCopy((CloudBlockBlob)srcBlob);

            }

        else if (srcBlob.BlobType == BlobType.AppendBlob){

           CloudAppendBlob dstblob = contDst.GetAppendBlobReference(dstpath);

           dstblob.StartCopy((CloudAppendBlob)srcBlob);

           }

        else if (srcBlob.BlobType == BlobType.PageBlob){

          CloudPageBlob dstblob = contDst.GetPageBlobReference(dstpath);

          dstblob.StartCopy((CloudPageBlob)srcBlob);

          }

       }

     }

  }

}

“@

Add-Type ReferencedAssemblies $reflib TypeDefinition $Source -Language CSharp PassThru

[ns.copyfn1]::Copy_bw_SA_Blobs_Test(“acc_key1”, “acc_key2”, “storage_acc1”, “storage_acc2”

, src_container_name, dest_container_name, “sales/2017/Jan”

, “sales/2017/backup/”)

 

With this inline C# code, we were able to optimize the copy time duration and also able to mitigate the  performance degrade over a heavy loop.

Let me know if this helps in someway or issue with this.

2017-04-17 Posted by | .NET, Azure, C#, Powershell | | Leave a comment

[Azure Storage] How to call Storage API’s from Powershell (without SDK)

Recently I had this ask from a partner who wanted sample code for making REST API call to storage without our SDK’s route. Though we have array of SDK’s supporting many languages, still he wanted to make clean REST calls without SDK’s. It some time to understand and create this proof of concept. So sharing here to easy reference. Hope this helps in some way..

$accountname=“your_storage_accname”

$key = “acc_key”

$container=“container_name”

#file to create

$blkblob=“samplefile.log”

$f = “C:\temp\samplefile.log”

$BlobOperation = PUT

$body = (Get-Content -Path $f -Raw)

$filelen = $body.Length

#added this per comments in the below blog post.

$filelen = (Get-ChildItem -File $f).Length

$RESTAPI_URL = “https://$accountname.blob.core.windows.net/$container/$blkblob;

$date=(Get-Date).ToUniversalTime()

$datestr=$date.ToString(“R”);

$datestr2=$date.ToString(“s”)+“Z”;

$strtosign = $BlobOperation`n`n`n$filelen`n`n`n`n`n`n`n`n`nx-ms-blob-type:BlockBlob`nx-ms-date:$datestr`nx-ms-version:2015-04-05`n/”

$strtosign = $strtosign + $accountname + “/”

$strtosign = $strtosign + $container

$strtosign = $strtosign + “/” +$blkblob

 

write-host $strtosign

 

[byte[]]$dataBytes = ([System.Text.Encoding]::UTF8).GetBytes($strtosign)

$hmacsha256 = New-Object System.Security.Cryptography.HMACSHA256

$hmacsha256.Key = [Convert]::FromBase64String($key)

$sig = [Convert]::ToBase64String($hmacsha256.ComputeHash($dataBytes))

$authhdr = “SharedKey $accountname`:$sig

 

write-host $authhdr

 

$RequestHeader = New-Object “System.Collections.Generic.Dictionary[[String],[String]]”

 

$RequestHeader.Add(“Authorization”, $authhdr)

$RequestHeader.Add(“x-ms-date”, $datestr)

$RequestHeader.Add(“x-ms-version”, “2015-04-05”)

$RequestHeader.Add(“x-ms-blob-type”,“BlockBlob”)

 

#create a new PS object to hold the response JSON

$RESTResponse = New-Object PSObject;

$RESTResponse = (Invoke-RestMethod -Uri $RESTAPI_URL -Method put -Headers $RequestHeader -InFile $f);

 

write-host $RESTResponse

write-host “# Success !!! uploaded the file >>” $RESTAPI_URL

————————————————————————————————————————————————————-

$accountname=“your_storage_accname”

$key = “acc_key”

$container=“container_name”

$blkblob=“file_for_deletion”

$BlobOperation = DELETE

 

$RESTAPI_URL = “https://$accountname.blob.core.windows.net/$container/$blkblob;

$date=(Get-Date).ToUniversalTime()

$datestr=$date.ToString(“R”);

$datestr2=$date.ToString(“s”)+“Z”;

 

$strtosign = $BlobOperation`n`n`n`n`n`n`n`n`n`n`n`nx-ms-blob-type:BlockBlob`nx-ms-date:$datestr`nx-ms-version:2015-04-05`n/”

 

$strtosign = $strtosign + $accountname + “/”

$strtosign = $strtosign + $container

$strtosign = $strtosign + “/” +$blkblob

write-host $strtosign

 

[byte[]]$dataBytes = ([System.Text.Encoding]::UTF8).GetBytes($strtosign)

$hmacsha256 = New-Object System.Security.Cryptography.HMACSHA256

$hmacsha256.Key = [Convert]::FromBase64String($key)

$sig = [Convert]::ToBase64String($hmacsha256.ComputeHash($dataBytes))

$authhdr = “SharedKey $accountname`:$sig

write-host $authhdr

 

$RequestHeader = New-Object “System.Collections.Generic.Dictionary[[String],[String]]”

$RequestHeader.Add(“Authorization”, $authhdr)

$RequestHeader.Add(“x-ms-date”, $datestr)

$RequestHeader.Add(“x-ms-version”, “2015-04-05”)

$RequestHeader.Add(“x-ms-blob-type”,“BlockBlob”)

$RESTResponse = New-Object PSObject;

write-host $RESTAPI_URL

 

$RESTResponse = (Invoke-RestMethod -Uri $RESTAPI_URL -Method Delete -Headers $RequestHeader);

 

write-host “# Success !!! deleted the input file >>” $RESTAPI_URL

reference:-

https://dzone.com/articles/examples-windows-azure-storage

https://docs.microsoft.com/en-us/azure/storage/storage-introduction#storage-apis-libraries-and-tools

2017-04-06 Posted by | Azure, Powershell, Storage | | 3 Comments

[Azure CDN] How to Purge CDN content from C# code

Recently I had a chance to work on this ask where one of the developer wanted a sample code to purge the CDN content by hitting this REST API instead portal. Initially I followed this link but do not wanted to give my username and password as explained there for AAD authentication. I wanted to get my app registered with AAD as WebApp and then work with ClientID and Client Secret way to the get the bearer token for making the REST API call. It worked rightly after so many attempts, so would like to leave the steps here for easy reference.

Step1: Register your Web Application in AAD (portal.Azure.com) and get the Client ID and Secret generated.

8

Step2) Select the required permissions as below.

7

6

Step3) Add this application as a contributor to the CDN Endpoint which we wanted to purge.

CDN Profile > CDN Endpoint > Access Control (IAM) > Add our Web Application which is going to hit this endpoint to purge

10

9

Step4) Now run our sample code to get invoke purged call

image       

Step5) On success.

5

Sample C# code used for purge ( from WebApplication )

       public ActionResult Contact()
        {
            GetAccessTokenAndMakePurgeCall();

            ViewBag.Result = “Done – Purged all.. “;

            return View();
        }

        private static void GetAccessTokenAndMakePurgeCall()
        {
            string clientId = “e32a947c-136e-xxxxxx-15eed998b592”;
            string clientSecret = “Ga9y3/9wNklz3Ft/xxxxxxxxgqpNM5KZxxxXM=”;
            string uri = @”
https://management.azure.com/subscriptions/xxxxxx-xxxxx/resourcegroups/emptybincdnrg/providers/Microsoft.Cdn/profiles/bluehousecdn/endpoints/bhep/purge?api-version=2015-06-01″;

            var authenticationContext = new AuthenticationContext(“https://login.microsoftonline.com/microsoft.onmicrosoft.com”);
            ClientCredential clientCredential = new ClientCredential(clientId, clientSecret);
            Task<AuthenticationResult> resultstr = authenticationContext.AcquireTokenAsync(“
https://management.core.windows.net/”, clientCredential);

            WebClient client = new WebClient();
            //authentication using the Azure AD application
            var token = resultstr.Result.AccessToken;
            client.Headers.Add(HttpRequestHeader.Authorization, “Bearer ” + token);
            client.Headers.Add(“api-version:2015-06-01”);
            client.Headers.Add(“Content-Type”, “application/json”);

            var bodyText = string.Empty;

            //For individual files
            //dynamic content = new { ContentPaths = new List<string>() { “/1.jpg”, “/2.jpg” } };

            //bodyText = JsonConvert.SerializeObject(content);

            //For purge all (*.*)
            //bodyText = “{“ContentPaths”:[“/*”]}”;

            try
            {
                var result = client.UploadString(uri, bodyText);
            }
            catch (Exception ew)
            {
                //handle the exception here
            }
        }

 

On running fiddler, we can see the header details are formed correctly and getting the bearer token to proceed.

image

On success, you could see “202” in the response which means our purge request has been accepted.

image

 

<update:4/18>

How to purge the CDN content using Postman ( assuming you have the valid bearer token )

1

2

On success, you would see “202” – accepted.

</update>

Hope this helps.

 

<Disclaimer> 

              a) This sample code provided here is for the purpose of illustration only and is not intended to be used in a production environment as it is.

              b) ContentPaths should be carefully checked. The path “/*” should be used only when you want to purge all the contents otherwise specify the resource path.

</Disclaimer> 

2017-04-01 Posted by | .NET, AAD, Azure, CDN | | 3 Comments

   

%d bloggers like this: