Building a Scalable Web Crawling and Indexing Pipeline with Azure storage and AI Search
In the ever-evolving world of data management, keeping search indexes up-to-date with dynamic data can be challenging. Traditional approaches, such as manual or scheduled indexing, are resource-intensive, delay-prone, and difficult to scale. Azure Blob Trigger combined with an AI Search Indexer offers a cutting-edge solution to overcome these challenges, enabling real-time, scalable, and enriched data indexing. This blog explores how Blob Trigger, integrated with Azure Cognitive Search, transforms the indexing process by automating workflows and enriching data with AI capabilities. It highlights the step-by-step process of configuring Blob Storage, creating Azure Functions for triggers, and seamlessly connecting with an AI-powered search index. The approach leverages Azure's event-driven architecture, ensuring efficient and cost-effective data management.
Automating Data Management: Azure Storage Actions Overview
Azure Storage Data Management Solutions/Services: 1. Movement: An end-to-end experience to discover, plan and move data into Azure in a performant, cost effective, secure, and reliable way 2. Insights: Store and manage metrics and metadata enabling deep insights into the data estate 3. Actions: Flexible, scalable, serverless platform to effortlessly process data for data management, protection, security and governance Big Data Challenges: Organizations must manage ever-increasing data volumes Data Management Data Movement Tagging & Classification Security and Access Control Data Protection Orchestration Customer challenges: Customers have increasingly large volumes of data in hundreds of storage accounts with billions of objects Challenging to process millions of objects for bulk operations Lifecycle management, data protection, object tagging and security operations require increasing complexity Out of box policies in storage can be constricted and extensibility is limited Introducing Azure Storage Actions A fully managed platform that helps you automate data management tasks.Process billions of blobs in your storage account effortlessly. Supports Azure Blob Storage and Azure Data Lake Storage How Storage Actions Works? Event-Condition-Action framework Schedule-based and on-demand execution Conditional processing of blobs based on blob properties Use native blob operations as actions on the blob Serverless Fully managed infrastructure Deploy in minutes – eliminates need for any complex software or infrastructure Auto-scales with your storage No-code composition & simplified management Use clicks to compose tasks Easily apply tasks to multiple storage accounts Monitor task execution across your storage execution with aggregate metrics and drilldowns Storage Actions Overview Data Protection – blob immutability, legal holds and blob expiry Cost optimization – tiering or deleting blobs Managing blob tags Undelete blobs Copy blobs, folder operations Key Concepts How to start with Storage Actions: Login to Portal: Search for Azure Storage Actions Create a task à Define Conditions: [[and(endsWith(Name, 'pdf'), equals(BlobType, 'BlockBlob'))]] The query [[and(endsWith(Name, 'pdf'), equals(BlobType, 'BlockBlob'))]] is a logical expression used to filter and retrieve specific items from a dataset. Here's a breakdown of its components: endsWith(Name, 'pdf'): This part of the query checks if the Name attribute of an item ends with the string 'pdf'. Essentially, it filters items whose names end with .pdf, indicating that they are PDF files. equals(BlobType, 'BlockBlob'): This part of the query checks if the BlobType attribute of an item is equal to 'BlockBlob'. This is used to filter items that are of the type BlockBlob, which is a type of storage blob in cloud storage systems. and(...): The and operator combines the two conditions above. It ensures that only items meeting both criteria are retrieved. In other words, the query will return items that are PDF files (Name ends with .pdf) and are of the type BlockBlob. In summary, this query is used to find items that are PDF files stored as BlockBlob in a dataset. If above query verified, it would set tags process = true, and set blob immutability policy to locked Create Assignment: Once task will run it will create report which will get added in Storage account container Example Use Cases Retention Management: Automatically manage the retention and expiry durations of audio files using a combination of index tags and creation times 4. Version History Management: Manage the retention and lifecycle of datasets using metadata and tags for optimal protection and cost 4. One-off Processing: Define tasks to rehydrate large datasets from the archive tier, reset tags on part of a dataset, or clean-up redundant and outdated datasets
Holding forensic evidence: The role of hybrid cloud in successful preservation and compliance
Disclaimer: The following is a post authored by our partner Tiger Technology. Tiger Technology has been a valued partner in the Azure Storage ecosystem for many years and we are happy to have them share details on their innovative solution! Police departments worldwide are grappling with a digital explosion. From body camera footage to social media captures, the volume and variety of evidence have surged, creating a storage, and management challenge like never before. A single police department needing to store 2–5 petabytes of data—and keep some of it for 100 years. How can they preserve the integrity of this data, make it cost-effective, and ensure compliance with legal requirements? The answer lies in hybrid cloud solutions, specifically Microsoft Azure Blob Storage paired with Tiger Bridge. These solutions are empowering law enforcement to manage, and store evidence at scale, without disrupting workflows. But what exactly is hybrid cloud, and why is it a game-changer for digital evidence management? What is a hybrid cloud? A hybrid cloud combines public or private cloud services with on-premises infrastructure. It gives organizations the flexibility to mix, and match environments, allowing them to choose the best fit for specific applications, and data. This flexibility is especially valuable in highly regulated industries like law enforcement, where strict data privacy, and compliance rules govern how evidence is stored, processed, and accessed. Hybrid cloud also facilitates a smoother transition to public cloud solutions. For instance, when a data center reaches capacity, hybrid setups allow agencies to scale dynamically while maintaining control over their most sensitive data. It’s not just about storage—it's about creating a robust, compliant infrastructure for managing enormous volumes of evidence. What makes digital evidence so complex? Digital evidence encompasses any information stored or transmitted in binary form that can be used in court. It includes computer hard drives, phone records, social media posts, surveillance footage, etc. The challenge isn’t just collecting this data—it’s preserving its integrity. Forensic investigators must adhere to strict chain-of-custody protocols to prove in court that the evidence: Is authentic and unaltered, Has been securely stored with limited access, Is readily available when needed. With the surge in data volumes and complexity, traditional storage systems often fall short. That’s where hybrid cloud solutions shine, offering scalable, secure, and cost-effective options that keep digital evidence admissible while meeting compliance standards. The challenges police departments face Digital evidence is invaluable. Storing and managing it is a challenging task, and requires dealing with several aspects: Short-term storage problems The sheer scale of data can overwhelm local systems. Evidence must first be duplicated using forensic imaging to protect the original file. But housing these duplicates, especially with limited budgets, strains existing resources. Long-term retention demands In some jurisdictions, evidence must be retained for decades—sometimes up to a century. Physical storage media, like hard drives or SSDs, degrade over time and are expensive to maintain. Transitioning this data to cloud cold storage offers a more durable and cost-effective solution. Data integrity and legal admissibility Even the slightest suspicion of tampering can render evidence inadmissible. Courts require robust proof of authenticity and integrity, including cryptographic hashes and digital timestamps. Failing to maintain a clear chain of custody could jeopardize critical cases. Solving the storage puzzle with hybrid cloud For law enforcement agencies, managing sensitive evidence isn't just about storage—it's about creating a system that safeguards data integrity, ensures compliance, and keeps costs under control. Traditional methods fall short in meeting these demands as the volume of digital evidence continues to grow. This is where hybrid cloud technology stands out, offering a powerful combination of on-premises infrastructure and cloud capabilities. Microsoft Azure, a leader in cloud solutions, brings critical features to the table, ensuring evidence remains secure, accessible, and compliant with strict legal standards. But storage alone isn't enough. Efficient file management is equally crucial for managing vast datasets while maintaining workflow efficiency. Tools like Tiger Bridge complement Microsoft Azure by bridging the gap between local and cloud storage, adding intelligence and flexibility to how evidence is preserved and accessed. Microsoft Azure Blob Storage Azure Blob Storage is massively scalable and secure object storage. For the purposes of law enforcement, among other features, it offers: Automatic Tiering: Automatically moves data between hot and cold tiers, optimizing costs, Durability: Up to sixteen 9s (99.99999999999999%) of durability ensures data integrity for decades. Metadata management: Add custom tags or blob indexes, such as police case classifications, to automate retention reviews. Microsoft Azure ensures evidence is secure, accessible, and compliant with legal standards. Tiger Bridge: Smart File Management Tiger Bridge enhances Microsoft Azure’s capabilities by seamlessly integrating local and cloud storage with powerful features tailored for forensic evidence management. Tiger Bridge is a software-only solution that integrates seamlessly with Windows servers. It handles file replication, space reclaiming, and archiving—all while preserving existing workflows and ensuring data integrity and disaster recovery. With Tiger Bridge, police departments can transition to hybrid cloud storage without adding hardware or altering processes. Data replication Tiger Bridge replicates files from on-premises storage to cloud storage, ensuring a secure backup. Replication policies run transparently in the background, allowing investigators to work uninterrupted. Files are duplicated based on user-defined criteria, such as priority cases or evidence retention timelines. Space reclamation Once files are replicated to the cloud, Tiger Bridge replaces local copies with “nearline” stubs. These stubs look like the original files but take up virtually no space. When a file is needed, it’s automatically retrieved from the cloud, reducing storage strain on local servers. Data archiving For long-term storage, Tiger Bridge moves files from hot cloud tiers to cold and / or archive storage. Files in the archive tier are replaced with "offline" stubs. These files are not immediately accessible but can be manually retrieved and rehydrated when necessary. This capability allows law enforcement agencies to save on costs while still preserving access to critical evidence. Checksum for data integrity On top of strong data integrity and data protection features already built-in in Azure Storage Blob service, Tiger Bridge goes a step further in ensuring data integrity by generating checksums for newly replicated files. These cryptographic signatures allow agencies to verify that files in the cloud are identical to the originals stored on premises. This feature is essential for forensic applications, where the authenticity of evidence must withstand courtroom scrutiny. Data integrity verification is done during uploads and retrievals, ensuring that files remain unaltered while stored in the cloud. For law enforcement, checksum validation provides peace of mind, ensuring that evidence remains admissible in court and meets strict regulatory requirements Disaster Recovery In the event of a local system failure, Tiger Bridge allows for immediate recovery. All data remains accessible in the cloud, and reinstalling Tiger Bridge on a new server re-establishes access without needing to re-download files. A real-life scenario Imagine a police department dealing with petabytes of video evidence from body cameras, surveillance footage, and digital device extractions. A simple, yet effective typical real-life scenario follows the similar patterns: Investigators collect and image evidence files, Tiger Bridge replicates this data to Azure Blob Storage, following predefined rules, Active cases remain in Azure’s hot tier, while archival data moves to cost-effective cold storage, Metadata tags in Azure help automate case retention reviews, flagging files eligible for deletion. This approach ensures evidence is accessible when needed, secure from tampering, and affordable to store long-term. The results speak for themselves. Adopting a hybrid cloud strategy delivers tangible benefits: Operational efficiency: Evidence is readily accessible without the need for extensive hardware investments and maintenance. Cost savings: Automating data tiering reduces storage costs while maintaining accessibility. Workflow continuity: Investigators can maintain existing processes with minimal disruption. Enhanced compliance: Robust security measures and chain-of-custody tracking ensure legal standards are met. A future-proof solution for digital forensics As digital evidence grows in both volume and importance, police organizations must evolve their storage strategies. Hybrid cloud solutions like Azure Blob Storage and Tiger Bridge offer a path forward: scalable, secure, and cost-effective evidence management designed for the demands of modern law enforcement. The choice is clear: Preserve the integrity of justice by adopting tools built for the future. About Tiger Technology Tiger Technology helps organizations with mission-critical deployments optimize their on-premises storage and enhance their workflows through cloud services. The company is a validated ISV partner for Microsoft in three out of five Azure Storage categories: Primary and Secondary Storage; Archive, Backup and BCDR, and Data Governance, Management, and Migration. Tiger Bridge SaaS offering on Azure Marketplace is an Azure benefit-eligible, data management software enabling seamless hybrid cloud infrastructure. Installed in the customer’s on-premises or cloud environment, Tiger Bridge intelligently connects file data across file and object storage anywhere for data lifecycle management, global file access, Disaster Recovery, data migration and access to insights. Tiger Bridge supports all Azure Blob Storage tiers, including cold and archive tiers for long-term archival of data. Read more by Tiger Technology on the Tech Community Blog: Modernization through Tiger Bridge Hybrid Cloud Data Services On-premises-first hybrid workflows in healthcare. Why start with digital pathology?
Announcing General Availability of Next generation Azure Data Box Devices
Today, we’re excited to announce the General Availability of Azure Data Box 120 and Azure Data Box 525, our next-generation compact, NVMe-based Data Box devices. These devices are currently available for customers to order in the US, US Gov, Canada, EU and the UK Azure regions, with broader availability coming soon. Since the preview announcement at Ignite '24, we have successfully ingested petabytes of data, encompassing multiple orders serving customers across various industry verticals. Customers have expressed delight over the reliability and efficiency of the new devices with up to 10x improvement in data transfer rates, highlighting them as a valuable and essential asset for large-scale data migration projects. These new device offerings reflect insights gained from working with our customers over the years and understanding their evolving data transfer needs. They incorporate several improvements to accelerate offline data transfers to Azure, including: Fast copy - Built with NVMe drives for high-speed transfers and improved reliability and support for faster network connections Ease of use - larger capacity offering (525 TB) in a compact form-factor for easy handling Resilient - Ruggedized devices built to withstand rough conditions during transport Secure - Enhanced physical, hardware and software security features Broader availability – Presence planned in more Azure regions, meeting local compliance standards and regulations What’s new? Improved Speed & Efficiency NVMe-based devices offer faster data transfer rates, providing a 10x improvement in data transfer speeds to the device as compared to previous generation devices. With a dataset comprised of mostly large (TB-sized) files, on average half a petabyte can be copied to the device in under two days. High-speed transfers to Azure with data upload up to 5x faster for medium to large files, reducing the lead time for your data to become accessible in the Azure cloud. Improved networking with support for up to 100 GbE connections, as compared to 10 GbE on the older generation of devices. Two options with usable capacity of 120 TB and 525 TB in a compact form factor meeting OSHA requirements. Devices ship the next day air in most regions. Learn more about the performance improvements on Data Box 120 and Data Box 525. Enhanced Security The new devices come with several new physical, hardware and software security enhancements. This is in addition to the built in Azure security baseline for Data Box and Data Box service security measures currently supported by the service. Secure boot functionality with hardware root of trust and Trusted Platform Module (TPM) 2.0. Custom tamper-proof screws and built-in intrusion detection system to detect unauthorized device access. AES 256-bit BitLocker software encryption for data at rest is currently available. Hardware encryption via the RAID controller, which will be enabled by default on these devices, is coming soon. Furthermore, once available, customers can enable double encryption through both software and hardware encryption to meet their sensitive data transfer requirements. These ISTA 6A compliant devices are built to withstand rough conditions during shipment while keeping both the device and your data safe and intact. Learn more about the enhanced security features on Data Box 120 and Data Box 525. Broader Azure region coverage A recurring request from our customers has been wider regional availability of higher-capacity devices to accelerate large migrations. We’re happy to share that Azure Data Box 525 will be available across US, US Gov, EU, UK and Canada with broader presence in EMEA and APAC regions coming soon. This marks a significant improvement in the availability of a large-capacity device as compared to the current Data Box Heavy which is available only in the US and Europe. What our customers have to say For the last several months, we’ve been working directly with our customers of all industries and sizes to leverage the next generation devices for their data migration needs. Customers love the larger capacity with form-factor familiarity, seamless set up and faster copy. “We utilized Azure Data Box for a bulk migration of Unix archive data. The data, originating from IBM Spectrum Protect, underwent pre-processing before being transferred to Azure blobs via the NFS v4 protocol. This offline migration solution enabled us to efficiently manage our large-scale data transfer needs, ensuring a seamless transition to the Azure cloud. Azure Data Box proved to be an indispensable tool in handling our specialized migration scenario, offering a reliable and efficient method for data transfer.” – ST Microelectronics Backup & Storage team “This new offering brings significant advantages, particularly by simplifying our internal processes. With deployments ranging from hundreds of terabytes to even petabytes, we previously relied on multiple regular Data Box devices—or occasionally Data Box Heavy devices—which required extensive operational effort. The new solution offers sizes better aligned with our needs, allowing us to achieve optimal results with fewer logistical steps. Additionally, the latest generation is faster and provides more connectivity options at data centre premises, enhancing both efficiency and flexibility for large-scale data transfers.” - Lukasz Konarzewski, Senior Data Architect, Commvault “We have had a positive experience overall with the new Data Box devices to move our data to Azure Blob storage. The devices offer easy plug and play installation, detailed documentation especially for the security features and good data copy performance. We would definitely consider using it again for future large data migration projects.” – Bas Boeijink, Cloud Engineer, Eurofiber Cloud Infra Upcoming changes to older SKUs availability Note that in regions where the next-gen devices are available, new orders for Data Box 80 TB and Data Box Heavy devices cannot be placed post May 31, 2025. We will however continue to process and support all existing orders. Order your device today! The devices are currently available for customers to order in the US, Canada, EU, UK, and US Gov Azure regions. We will continue to expand to more regions in the upcoming months. Azure Data Box provides customers with one of the most cost-effective solutions for data migration, offering competitive pricing with the lowest cost per TB among offline data transfer solutions. You can learn more about the pricing across various regions by visiting our pricing page. You can use the Azure portal to select the requisite SKU suitable for your migration needs and place the order. Learn more about the all-new Data Box devices here. We are committed to continuing to deliver innovative solutions to lower the barrier for bringing data to Azure. Your feedback is important to us. Tell us what you think about the new Azure Data Box devices by writing to us at [email protected] – we can’t wait to hear from you.Azure Blob Storage SFTP: General Availability of ACLs (Access Control Lists) of local users
We are excited to announce the general availability of ACLs (Access Control Lists) for Azure Blob Storage SFTP local users. ACLs make it simple and intuitive for administrators to manage fine-grained access control to blobs and directories for Azure Blob Storage SFTP local users. Azure Blob Storage SFTP Azure Blob storage supports the SSH File Transfer Protocol (SFTP) natively. SFTP on Azure Blob Storage lets you securely connect to and interact with the contents of your storage account by using an SFTP client, allowing you to use SFTP for file access, file transfer, and file management. Learn more here. Azure Blob Storage SFTP is used by a significant number of our customers, who have shared overwhelmingly positive feedback. It eliminates the need for third-party or custom SFTP solutions involving cumbersome maintenance steps such as VM orchestration. Local users Azure Blob Storage SFTP utilizes a new form of identity management called local users. Local users must use either a password or a Secure Shell (SSH) private key credential for authentication. You can have a maximum of 25,000 local users for a storage account. Learn more about local users here. Access Control for local users There are two ways in which access control can be attained for local users. 1. Container permissions By using container permissions, you can choose which containers you want to grant access to and what level of access you want to provide (Read, Write, List, Delete, Create, Modify Ownership, and Modify Permissions). Those permissions apply to all directories and subdirectories in the container. Learn more here. 2. ACLs for local users What are ACLs? ACLs (Access Control Lists) let you grant "fine-grained" access, such as write access to a specific directory or file, which isn’t possible with Container Permissions. More fine-grained access control has been a popular ask amongst our customers, and we are very excited to make this possible now with ACLs. A common ACL use case is to restrict a user's access to a specific directory without letting that user access other directories within the same container. This can be repeated for multiple users so that they each have granular access to their own directory. Without ACLs, this would require a container per local user. ACLs also make it easier for administrators to manage access for multiple local users with the help of groups. Learn more about ACLs for local users here. How to set and modify the ACL of a file or a directory? You can set and modify the permission level of the owning user, owning group, and all other users of an ACL by using an SFTP client. You can also change the owning user or owning group of a blob or directory. These operations require 'Modify Permissions' and 'Modify Ownership' container permissions, respectively. Note: Owning users can now also modify the owning group and permissions of a blob or directory without container permissions. This is a new feature enhancement added during the General Availability phase of ACLs for local users. For any user that is not the owning user, container permissions are still required. Learn more here. These enhancements significantly improve the management and usability of Azure Blob Storage SFTP by providing more granular access control over the container model and extending customer options. Please reach out to [email protected] for feedback about SFTP for Azure Blob Storage. We look forward to your continued support as we strive to deliver the best possible solutions for your needs.Simplifying Kubernetes Data Protection with CloudCasa and Microsoft Azure Blob Storage
NOTE: This article is co-authored with our partner, CloudCasa. Ensuring reliable backups and fast recoveries in these hybrid setups is challenging. CloudCasa’s partnership with Microsoft Azure Blob Storage offers a robust solution to this challenge, delivering scalable, secure, and cost-effective data protection for Kubernetes cloud and hybrid cloud workloads. In this blog, we will look at how CloudCasa and Azure Blob solve common data protection challenges. We will explore how they simplify backups for Azure Kubernetes Service (AKS), offer centralized protection for hybrid clusters with Azure Arc, and give you a single view for managing VM and container backups. We will also cover how they help with disaster recovery in Kubernetes environments and how they push data to Azure Blob in an efficient manner. By the end, you will see how this partnership prepares your data protection strategy for the future. Seamless Integration with the Azure Ecosystem A good backup solution should not create more work. It should blend into the tools you already use. CloudCasa is purpose-built to integrate effortlessly with the Microsoft Azure ecosystem. This integration provides robust data protection for Kubernetes and hybrid cloud environments through: CloudCasa for AKS: Azure Kubernetes Service (AKS) is a popular way to run Kubernetes in Microsoft Azure. CloudCasa for AKS gives you an easy way to protect your containers, volumes, and metadata. You can set schedules and policies in CloudCasa to back up entire AKS clusters. If something goes wrong, you can quickly restore your workloads to the same or a different AKS cluster. Learn more about CloudCasa for AKS. CloudCasa for Azure Arc: Azure Arc extends Azure services to on-premises, multi-cloud, and edge environments. That means you can manage servers and Kubernetes clusters from a single point. CloudCasa for Azure Arc takes advantage of this central view. It lets you protect your Arc-enabled Kubernetes clusters wherever they run. Instead of juggling multiple backup tools for each environment, you can protect them all from a single place. Learn more about CloudCasa for Azure Arc. Azure Marketplace: Deploying and paying for new services often involves many steps. But CloudCasa is available on the Azure Marketplace, making it easy to start. You can sign up with a few clicks and have the charges appear on your Azure bill. This straightforward setup reduces friction and helps teams begin protecting their workloads right away. Supporting Hybrid Cloud and Kubernetes Workloads Modern organizations rarely keep all of their data in a single place. Some workloads might still run on-premises, while others run in different clouds. Kubernetes clusters can be scattered across these environments. Without a proper solution, managing backups for all these clusters can get messy. CloudCasa manages backups in a clever way. By default, it uses incremental backups whenever possible, which means it only moves the data that has changed since the last backup. This minimizes network usage and lowers storage costs. All these backups are stored in Azure Blob, which is both scalable and cost-effective. Unified Backup Management for Arc-Enabled Clusters When you connect your Kubernetes clusters to Azure Arc, you get a single view of all those clusters in the Azure portal. CloudCasa fits into this model by letting you create and apply backup policies across your hybrid environment. Instead of setting backup rules cluster-by-cluster, you can handle them from one place. This unification saves time and helps you avoid errors. A good backup is only valuable if you can recover from it quickly and without hassle. CloudCasa simplifies this process. You can restore your containers, PVs (Persistent Volumes), and cluster configurations back to their original places. This limits downtime and keeps your business moving. If your cluster is no longer available, you can also restore to a new location, including AKS, to keep your workloads running. Streamlining Kubernetes Disaster Recovery Disaster recovery (DR) is a top priority for many organizations. Traditional DR methods might involve copying data to a secondary site or region, then running complex scripts to bring everything back online. In a Kubernetes world, DR can be simpler if done the right way. Containers let you package your applications and dependencies in a portable format. If you have regular backups, you can quickly spin up your applications on another cluster if one site goes down. CloudCasa focuses on container-aware backups, so your entire environment—services, pods, configuration, and persistent data—can be restored. You don’t have to worry about missing parts of your cluster during recovery. If disaster strikes, you might need to create a brand-new cluster in Azure Kubernetes Service. CloudCasa makes this smoother. You can use your existing backups to set up a cluster in AKS, restoring namespaces, deployments, and volumes. This approach gives you a quick path to recovery without diving into manual steps or complicated scripts. Short-Term Recovery and Long-Term Archiving Azure Blob storage can keep multiple versions of your backups. This is helpful if you need to recover something quickly, like a file that was deleted by mistake. You can also use Azure Blob for long-term archiving, storing older backups at lower cost in access tiers like Cool or Archive. That means you meet compliance needs without breaking the bank, and you can still restore data if needed down the line. Sending backup data to the cloud must be efficient, both in terms of resource usage and network costs. If your backups overload your network or compute resources, it can slow down your day-to-day operations. CloudCasa is built to avoid these problems. Lightweight Agent Technology CloudCasa has a lightweight agent that runs in your Kubernetes clusters. This agent takes care of backup tasks without draining your compute and network resources. The agent only captures what it needs. With incremental backups, you don’t send the same data over and over again, which keeps your workload overhead low. CloudCasa includes metrics that show how much data is being backed up and how quickly. You can also see if there are any issues during the backup process. These insights help you spot bottlenecks and optimize your data protection approach. You can also control how often backups happen, which helps reduce unplanned spikes in storage costs or network usage. Cost-Effective Scalability Azure Blob follows a pay-as-you-go model. You only pay for what you use, whether it’s the size of stored data or how often you read and write that data. As your Kubernetes clusters grow or you add more clusters, CloudCasa and Azure Blob scale with you. There is no need to worry about running out of space or making large upfront investments in storage hardware. The technology landscape keeps changing. Businesses need solutions that can adapt to new platforms, new workloads, and new regulations. CloudCasa and Azure Blob offer that flexibility. Scalable Data Protection for Hybrid and Cloud-Native Environments Today you might be running a few clusters on-premises and a couple in the cloud. Next year, you might expand to more regions or adopt new container platforms. CloudCasa’s agent-based design and Azure Blob’s elastic storage model are built to handle this. You can protect more clusters as you go, without complex setup or hardware upgrades. In many industries, regulations require you to keep data for a certain period and show that it’s stored in a durable, secure way. Azure Blob provides high durability with multiple copies of your data in different locations. CloudCasa’s backups are encrypted in transit and at rest. This means you can meet compliance needs and still have control over your data retention policies. Advanced Recovery Capabilities As you expand, your recovery needs may become more complex. You may have to recover apps across different regions or clouds. CloudCasa gives you the option to restore to different clusters, whether they are on Azure or on another provider. This means you can change course when needed, without being tied to a single environment. And because Azure Blob keeps your data in a secure and highly available way, you can trust that it will be there when you need it. Conclusion Protecting Kubernetes workloads can be simpler than you think. With CloudCasa and Microsoft Azure Blob, you get a scalable and secure backup solution that works across hybrid and cloud-native environments. You also have a unified tool for both container and VM backups, which reduces management overhead. In case of disaster, CloudCasa helps you recover quickly to AKS or your original environment. And because everything is stored in Azure Blob, you benefit from its pay-as-you-go model and global infrastructure. By adopting this integrated approach, you can focus on delivering applications and services, rather than juggling multiple backup systems. CloudCasa makes it easy to set backup schedules and policies, track performance, and restore data wherever you run your workloads. You gain peace of mind knowing your data is protected and can be recovered quickly, without complicated scripts or risky manual steps. Next steps To learn more, visit CloudCasa for AKS and CloudCasa for Azure Arc on the Azure Marketplace, explore CloudCasa.io, or at [email protected] to get started. Additionally, we have a demo video showcasing the capabilities of both AKS and Azure Arc together, providing a comprehensive overview of how these solutions can benefit your organization.
Microsoft Purview Protection Policies for Azure Data Lake & Blob Storage Available in All Regions
Organizations today face a critical challenge: ensuring consistent and automated data governance across rapidly expanding data estates. Driven by the growth of AI and the increasing reliance on vast data volumes for model training, Chief Data Officers (CDOs) and Chief Information Security Officers (CISOs) must prevent unintentional exposure of sensitive data (PII, credit card information) while adhering to data and legal regulations. Many organizations rely on Azure Blob Storage and ADLS for storing vast amounts of data, offering scalable, secure, and highly available cloud storage solutions. While solutions like RBAC (role-based access control), ABAC (attribute-based access control), and ACLs (Access Control Lists) offer secure ways to manage data access, they can operate on metadata such as file paths, tags, or container names. These mechanisms are effective for implementing restrictive data governance by controlling who can access specific files or containers. However, there are scenarios were implementing automatic access controls based on the sensitivity of the content itself is necessary. For example, identifying and protecting sensitive information like credit card numbers within a blob requires more granular control. Ensuring that sensitive content is restricted to specific roles and applications across the organization is crucial, especially as enterprises focus on building new applications and infusing AI into current solutions. This is where integrated solutions like Microsoft Information Protection (MIP) come into play. Microsoft Information Protection (MIP) protection policies provide a solution by enabling organizations to scan and label data based on the content stored in the blob. This allows for applying access controls directly related to the data asset content across storage accounts. By eliminating the need for in-house scanning and labeling, MIP streamlines compliance and helps in applying consistent data governance using a centralized solution. The Solution: Microsoft Purview Information Protection (MIP) Protection Policies for Governance & Compliance Microsoft Purview Information Protection (MIP) provides an efficient and centralized approach to data protection by automatically restricting access to storage data assets based on sensitivity labels discovered through automated scanning and leveraging Protection policies (learn more). This feature builds upon Microsoft Purview's existing capability (learn more) to scan and label sensitive data assets, ensuring robust data protection. This not only enhances data governance but also ensures that data is managed in a way that protects sensitive information, reducing the risk of unauthorized access and maintaining the security and trust of customers. Enhancing Data Governance with MIP Protection policies: Contoso, a multinational corporation, handles large volumes of data stored in Azure Storage (Blob/ADLS). Different users, such as financial auditors, legal advisors, compliance officers, and data analysts, need access to different blobs in the Storage account. These blobs are updated daily with new content, and there can be sensitive data across these blobs. Given the diverse nature of the stored data, Contoso needed an access control method that could restrict access based on data asset sensitivity. For instance, data analysts access the blob named "logs" where log files are uploaded. If these files contain PII or financial data, which should only be accessed by financial officers, the access permissions need to be dynamically updated based on the changing sensitivity of the stored data. MIP protection policies can address this challenge efficiently by automatically limiting access to data based on sensitivity labels found through automated scanning. Key Benefits: Auto-labelling: Automatically apply sensitivity labels to Azure Storage based on detection of sensitive information types. Automated Protection: Automatically restrict access to data with specific sensitivity labels, ensuring consistent data protection. Storage Data Owners can selectively enable specific storage accounts for policy enforcement, providing flexibility and control. Like a protection policy that restricted access to data labeled as "Highly Confidential" to only specific groups or users. For instance, blobs labeled with "logs" were accessible only to data analysts. With MIP, the labels are updated based on content changes, and the protection policy can deny access if the content if any “Highly Confidential” data is identified. Enterprise-level Control: Information Protection policies are applied to blobs and resource sets, ensuring that only authorized Azure Entra ID users or M365 user groups can access sensitive data. Unauthorized users will be prevented from reading the blob or resource set. Centralized Policy Management: Create, manage, and enforce protection policies across Azure Storage from a single, unified interface in Microsoft Purview. Enterprise admins have granular control over which storage accounts enforce protection coverage based on the account’s sensitivity label. By using Microsoft Purview Information Protection (MIP) Protection Policies, Contoso was able to achieve secure and consistent data governance, and centralized policy management, effectively addressing their data security challenges Prerequisites Microsoft 365 E5 licenses and setup of pay as you go billing model. To understand pay as you go billing by assets protected, see the pay-as-you-go billing model. For information about the specific licenses required, see this information on sensitivity labels. Microsoft 365 E5 trial licenses can be attained for your tenant by navigating here from your environment. Getting Started The public preview of Protection Policies supports the following Azure Storage services: Azure Blob Storage Azure Data Lake Storage To enable Protection Policies for your Azure Storage accounts: Navigate to the Microsoft Purview portal> Information Protection card > Policies. Configure or use an existing sensitivity label in Microsoft Purview Information Protection that’s scoped to “Files & other data assets” Create an auto-labelling to apply a specific sensitivity label to scoped assets in Azure Storage based on Microsoft out-of-the-box sensitive info types detected. Run scans on assets for auto-labelling to apply. Create a protection policy and associate it with your desired sensitivity labels. Apply the policy to your Azure Blob Storage or ADLS Gen2 accounts. Limitations During the public preview, please note the following limitations: Currently a maximum of 10 storage accounts are supported in one protection policy, and they must be selected under Edit for them to be enabled. Changing pattern rules will re-apply labels on all storage accounts. During the public preview, there might be delays in label synchronization, which could prevent MIP policies from functioning effectively. If customer storage account enables CMK, the storage account MIP policy will not work. Next Steps With the Public Preview, MIP Protection policies is now available in all regions, and any storage account registered on the Microsoft Purview Data Map can create and apply protection policies to implement consistent data governance strategies across their data in Azure Storage. We encourage you to try out this feature and provide feedback. Your input is crucial in shaping this feature as we work towards general availability.
Control geo failover for ADLS and SFTP with unplanned failover.
We are excited to announce the General Availability of customer managed unplanned failover for Azure Data Lake Storage and storage accounts with SSH File Transfer Protocol (SFTP) enabled. What is Unplanned Failover? With customer managed unplanned failover, you are in control of initiating your failover. Unplanned failover allows you to switch your storage endpoints from the primary region to the secondary region. During an unplanned failover, write requests are redirected to the secondary region, which then becomes the new primary region. Because an unplanned failover is designed for scenarios where the primary region is experiencing an availability issue, unplanned failover happens without the primary region fully completing replication to the secondary region. As a result, during an unplanned failover there is a possibility of data loss. This loss depends on the amount of data that has yet to be replicated from the primary region to the secondary region. Each storage account has a ‘last sync time’ property, which indicates the last time a full synchronization between the primary and the secondary region was completed. Any data written between the last sync time and the current time may only be partially replicated to the secondary region, which is why unplanned failover may incur data loss. Unplanned failover is intended to be utilized during a true disaster where the primary region is unavailable. Therefore, once completed, the data in the original primary region is erased, the account is changed to locally redundant storage (LRS) and your applications can resume writing data to the storage account. If the previous primary region becomes available again, you can convert your account back to geo-redundant storage (GRS). Migrating your account from LRS to GRS will initiate a full data replication from the new primary region to the secondary which has geo-bandwidth costs. If your scenario involves failing over while the primary region is still available, consider planned failover. Planned failover can be utilized in scenarios including planned disaster recovery testing or recovering from non-storage related outages. Unlike unplanned failover, the storage service endpoints must be available in both the primary and secondary regions before a planned failover can be initiated. This is because planned failover is a 3-step process that includes: (1) making the current primary read only, (2) syncing all the data to the secondary (ensuring no data loss), and (3) swapping the primary and secondary regions so that writes are now in the new region. In contrast with unplanned failover, planned failover maintains the geo-redundancy of the account so planned failback does not require a full data copy. To learn more about planned failover and how it works view, Public Preview: Customer Managed Planned Failover for Azure Storage | Microsoft Community Hub To learn more about each failover option and the primary use case for each view, Azure storage disaster recovery planning and failover - Azure Storage | Microsoft Learn How to get started? Getting started is simple, to learn more about the step-by-step process to initiate an unplanned failover review the documentation: Initiate a storage account failover - Azure Storage | Microsoft Learn Feedback If you have questions or feedback, reach out at [email protected]
