Author: Ozzy Brightrose

AWS Data and Analytics Use Case Partner

Posted on | by Ozzy Brightrose | Leave a Comment on AWS Data and Analytics Use Case Partner

Amazon Web Services is a broadly adopted cloud platform with over 175 cloud-based services for businesses of all types. APN partner works as the global partner program for technology and consulting businesses to find out the most reliable solutions to their needs. 

 

It’s a one-stop platform for all types of web-based businesses to develop, market and sell AWS solutions by offering top-notch supports and services as per their requirements. 

 

At present, there are tens of thousands of APN Partners across the globe with more than 90% of fortune 100 companies and the majority of fortune 500 companies utilize APN Partners solutions and services to see the desired growth in their respective businesses. 

 

Data & Analytics APN Partners 

The main aim behind working with the APN Partners is to shift up your business with effective migration to cloud platform. You can easily find out suitable APN Partners by using the AWS Partner Solutions Finder tool. This tool is designed for the customers to search and connect with the trusted and reliable APN Partners as per their needs. AWS Customer can make use of the AWS Partner Solutions Finder to find out the suitable APN Partner for handling workloads on the AWS platform. 

 

Basically, there are three types of categories available to find out the suitable APN Partners: 

 

  • Industry
  • AWS Product
  • Use Case

 

In this article, we will discuss some of the most trusted APN Partners based on Data & Analytics Use Cases on the AWS Platform: 

 

1. Cloudreach 

Cloudreach is a leading cloud service provider in the world headquarter in London. The company is known for providing top-notch services to businesses to monetize their business data and to see the desired growth with it. They offer a variety of services through their application, software and cloud-based services which includes all the industries and businesses.  

 

Cloudreach is an original partner of AWS environment that lets you find the right solution for your business requirements. You will be provided with proper guidelines to find out the suitable AWS solutions which can help elevate your business. Cloudreach was named 2019 UK & I APN Consulting Partner of the year.  

AWS Data and Analytics Use Case - Cloudreach

AWS Data and Analytics Use Case – Cloudreach

2. Deloitte  

When it comes to the digital transformation of your business, Deloitte comes first. It is an industry leader and a trusted name for providing professional services and solutions to businesses of all kinds. The company is headquartered in London and is functional in many countries across the globe. 

 

The company has over 244,000 professionals and specialists to simplify major issues of your businesses and convert them solutions by deep analysis. The market-leading team helps businesses and industries to see the ray of hope in the new digital era.  

 

If you are a casual learner, then you can also find out the reliable sources through Deloitte’s network to get started with a new career. They offer a variety of services which are there not only for the enterprises but for casual users also.  

AWS Data and Analytics Use Case - Deloitte

AWS Data and Analytics Use Case – Deloitte

3. Storm Reply  

Storm Reply is known for providing solutions to businesses and industries through effective designing and development strategies. The company holds Amazon Premier Consulting Partner Status to help significant customers run their applications and businesses on the AWS Platform. The company provides solutions through modern communication networks and digital media.  

 

With great experience in Cloud SaaS, IaaS and PaaS architecture, the company provides end-to-end services to its customers with full satisfaction. Cloud Reply supports important companies in Europe and across the world for the implementations of cloud-based systems and applications.  

AWS Data and Analytics Use Case - Storm Reply

AWS Data and Analytics Use Case – Storm Reply

 

4. Ecloudvalley  

 Ecloudvalley is the leading AWS Premium Consulting Partner to evolve businesses in the cloud by providing effective solutions. The company is one of the most trusted cloud advisors on the AWS Platform with over 400+ certificates. With the team of highly qualified and experienced specialists, they deliver the best of solutions to convert your physical business to cloud-based one.  

 

Ecloudvallye is known for providing services like cloud training, cloud migration, data solution, next-generation MSP and automated cloud management platform. They have served over 1000+ customers in the region and planning to expand their services across the globe in the coming months.  

AWS Data and Analytics Use Case - EcloudValley

AWS Data and Analytics Use Case – EcloudValley

5. Wipro  

Wipro is a trusted name in the technology world as it is a global company for providing technology, consulting and business process services for significant customers across the globe. The company is headquartered in Bangalore, India and offering cloud-based services to customers from across the globe.  

 

The company aims at delivering top-notch services by combining different technologies including robotics, cloud analytics, hyper-automation and more to help their clients adopt these technologies to make their businesses functional on the digital platform. 

 

Wipro is a leading APN Partner focuses on providing a complete AWS cloud service through Cloud services, Cybersecurity and Data Analysis. You can explore the range of solutions and pick the most reliable one that suits your business needs. 

AWS Data and Analytics Use Case - Wipro

AWS Data and Analytics Use Case – Wipro

APN training partner

AWS APN Training

AWS APN Training Use Case

Posted on | by Ozzy Brightrose

What is AWS APN Training?

The AWS Partner Network (APN) works as the global partner program for technology and consulting businesses. These businesses use Amazon Web Services to build solutions and serve their customers in better ways. The AWS Partner Network helps businesses to develop, market, and sell their AWS solutions by offering valuable business, technical, and marketing support.c

“There are tens of thousands of APN Partners across the globe. More than 90% of Fortune 100 companies and the majority of Fortune 500 companies utilize APN Partner solutions and services.”

Authorized AWS APN Training Partners

Working with APN partners will help you grow and scale your business by accelerating your migration process to the cloud. You can find APN Partners with the help of AWS Partner Solutions Finder. This tool help AWS customers to search and connect with trusted APN Partners as per their business requirement. Being an AWS customer, you can use AWS Partner Solutions Finder to discover an APN Partner that can help you design, migrate, manage, and optimize workloads on AWS.

You can find APN Partners based on three categories:

  • Industry
  • Use Case
  • AWS Product

In this blog, we will discuss some APN Partners based on AWS-authorized training use cases.

#1. MegazoneCloud

MegazoneCloud is Korea’s first AWS Premier Consulting Partner. This company was awarded ‘Partner of the Year’ for APAC and Korea for 2 consecutive years. Megazone has been successful in achieving various AWS competencies, including Marketing & Commerce, Migration, and Financial Services, and offers the most advanced cloud services to its clients. The company provides AWS solutions and value-added services to more than 1,500 customers in various fields, such as large companies, startups, and public institutions, etc.

AWS APN Training - Megazone Cloud

AWS APN Training – Megazone Cloud

 

#2. NORDCLOUD

Nordcloud has been offering cloud implementation and management across Europe since 2011. The company is helping Enterprise customers to migrate to the public cloud, transform and manage their current applications and build cloud-native solutions. From project-specific support to managed services, they can help you accelerate time to market, maximize your cost savings, and recognize your growth ambitions. They offer a comprehensive range of cloud services – from DevOps support and Kubernetes enablement to training and managed applications.

AWS APN Training - Nordcloud

AWS APN Training – Nordcloud

 

#3. CAPSiDE | NTT

CAPSiDE | NTT is trying its best to make the world more reliable with the help of design, automation & operation of critical digital platforms. They have a team of AWS experts who have nurtured their SysOps, DevOps & SecOps skills to provide expert cloud services as an extension of their customer’s business. Whether you are investigating, starting, or executing cloud adoption, CAPSiDE can help you, perfectly combining agility with stability.

You can connect with them as they design, deploy, and manage the cloud architecture round-the-clock and also ensure to meet your business requirements. They take complete responsibility for the migration of your business-critical applications to a cloud environment, focusing on scalability and performance while optimizing costs. The various other professional services offered by CAPSiDE | NTT are Managed Service Provider, Strategic / IT Consulting, Systems Integration, Training, and Assessment / Jumpstart Pkg.

AWS APN Training - Capside NTT

AWS APN Training – Capside NTT

#4. beSharp

beSharp is one of the most reliable APN Partners. The company has been successful in achieving several competencies, Well-Architected, DevOps, and Solution Provider, among others. With an experienced team of AWS experts, beSharp offers various services – from Cloud migrations to Cloud-Native Development, from DevOps to Managed Services with 24/7 customer support. You can trust beSharp for your business growth as beSharp takes care of every aspect of Cloud migration projects, from strategy to delivery, and operations.

They help companies to adopt revolutionary technological models, creating innovative projects that achieve their most ambitious goals. The cloud services offered by beSharp are AWS Professional services, AWS Managed services and AWS Native Development.

AWS APN Training - beSharp

AWS APN Training – beSharp

#5. Ecloudvalley

eCloudvalley is the 1st AWS Premier Consulting Partner in the Greater China Region (GCR). They are the trusted partners of over 800 customers and currently hold more than 400 AWS certifications. Their services are based on licensed original AWS services, including cloud storage, disaster backup, server hosting, server monitoring, and official AWS training. They also provide solutions to their customers for management consultant roles, including examining existing information security risks and system architecture weaknesses, improving system performance, and lowering corporate expenses on IT.

Some of the services offered by Ecloudvalley are:

  • Cloud Consultation
  • Cloud Migration
  • DevOps
  • Big Data
  • Managed Services
  • AWS Training
AWS APN Training - Ecloudvalley

AWS APN Training – Ecloudvalley

See Also

AWS machine learning use case

APN Partners

APN Consulting Partners

S3 to EC2 Data Transfer cost

S3 to EC2 Data Transfer Cost

Posted on | by Ozzy Brightrose | Leave a Comment on S3 to EC2 Data Transfer Cost

S3 to EC2 Data Transfer Cost

You will ultimately have to pay for all bandwidth going into or out of S3, excluding for the below listed:
• Data transfer in – from internet.
• Data transfer out – to an EC2 instance, if this instance lies in the same Region as that of your selected S3 bucket.
• Data transfer out to CloudFront.

The following pricing is based on data transfers into and out of S3.

Transfers occurring between S3 buckets or from S3 to whatever selected service while remaining in the same Region are free of charge.

You will need to pay an additional fee for whatever data transfers made through S3 Transfer Acceleration.

Region: US East (Ohio)

Price
Data Transfer IN To Amazon S3 From Internet
All data transfer in$0.00 per GB
Data Transfer OUT From Amazon S3 To Internet
Up to 1 GB / Month$0.00 per GB
Next 9.999 TB / Month$0.09 per GB
Next 40 TB / Month$0.085 per GB
Next 100 TB / Month$0.07 per GB
Greater than 150 TB / Month$0.05 per GB
Data Transfer OUT From Amazon S3 To
CloudFront$0.00 per GB
US East (N. Virginia)$0.01 per GB
Asia Pacific (Singapore)$0.02 per GB
Asia Pacific (Sydney)$0.02 per GB
US West (Los Angeles)$0.02 per GB
Canada (Central)$0.02 per GB
Europe (London)$0.02 per GB
Europe (Paris)$0.02 per GB
Asia Pacific (Tokyo)$0.02 per GB
AWS GovCloud (US-East)$0.02 per GB
Europe (Frankfurt)$0.02 per GB
Europe (Ireland)$0.02 per GB
Asia Pacific (Osaka-Local)$0.02 per GB
Asia Pacific (Seoul)$0.02 per GB
Asia Pacific (Mumbai)$0.02 per GB
Africa (Cape Town)$0.02 per GB
South America (Sao Paulo)$0.02 per GB
Middle East (Bahrain)$0.02 per GB
AWS GovCloud (US-West)$0.02 per GB
Asia Pacific (Hong Kong)$0.02 per GB
Europe (Stockholm)$0.02 per GB
US West (Northern California)$0.02 per GB
US West (Oregon)$0.02 per GB

Storage + bandwidth size: All file overhead included in it.

Data Transfer Out: Not the same as data received by an app if the connection is earlier terminated, like when you decide to make a request for a selected 10GB object and later on simply terminate the connection upon getting the 1st 2GB of data. S3 will jump to stop data streaming, in a non-instantaneous manner. In the stated example, Data Transfer Out is capable of being 3GB. Consequently, you are going to get billed for the 3GB of Data Transfer Out.

S3 to EC2 Data Transfer cost - transfer acceleration usage

S3 to EC2 Data Transfer cost – transfer acceleration usage

Transfer acceleration Usage

Pricing is according to which edge location utilized for accelerating the data transfer. The pricing for S3 Transfer Acceleration will be added to your Data Transfer pricing.

Data Transfer IN to Amazon S3 from the Internet:
Accelerated by AWS Edge Locations in the United States, Europe, and Japan$0.04 per GB
Accelerated by all other AWS Edge Locations$0.08 per GB
Data Transfer OUT from Amazon S3 to the Internet:
Accelerated by any AWS Edge Location$0.04 per GB
Data Transfer between Amazon S3 and another AWS region:
Accelerated by any AWS Edge Location$0.04 per GB

Every time people choose to go for the S3 Transfer Acceleration for the sake of uploading any specific object, it is going to get checked to see if the selected service is probably going to work in a faster pace than that of just the normal S3 transfer. In case it is determined that it won’t technically work in a faster speed than the normal S3 transfer process of the same object to the exact same chosen destination Region, you are not going to be charged for the use of this S3 Transfer Acceleration for this particular transfer, and might as well bypass the S3 Transfer Acceleration system for this specific upload.

 

AWS free usage tier

S3 to EC2 Data Transfer cost - free tier services by usage

S3 to EC2 Data Transfer cost – free tier services by usage

The Free Usage Tier, provides you with the opportunity to begin with getting to know Amazon S3 without having to pay a penny. As soon as you tend to sign-up, as a new AWS customer, you are going to get the following features:

– 5GB of S3 storage in the S3 Standard storage class

– 20,000 GET Requests

– 2,000 COPY, LIST, POST or PUT Requests

– 15GB of Data Transfer Out every single month for a period of one whole year

Your free tier usage is going to get calculated every single month across every single Region excluding the GovCloud Region and it is automatically going to be added to your billing information; Any unused monthly usage is not going to roll over, and some restrictions will apply.

s3 data transfer cost

s3 cost calculator

AWS S3 Pricing Model

S3 Data Transfer Pricing

S3 Data Transfer Pricing

Posted on | by Ozzy Brightrose | Leave a Comment on S3 Data Transfer Pricing

S3 Data Transfer Pricing

You must be aware of the fact that you will have to pay for every single bandwidth coming into and going out of your Amazon S3, excluding for the below mentioned:
• Data which gets transferred in from the internet.
• Data which gets transferred out to a specific EC2 instance, where the instance is located in the exact Region as that of the S3 bucket.
• Data which gets transferred out to CloudFront.

The pricing listed in the table below is in accordance with data which gets transferred “in” and “out” of your Amazon S3 across public Internet. Transfers taking place between S3 buckets or from S3 going to any specific service found within the same Region are cost free. You will additionally have to pay a fee for whatever data getting transferred through the use of S3 Transfer Acceleration.

 

Region: US East (Ohio) 

Data Transfer IN To Amazon S3 From InternetPrice
All data transfer in$0.00 per GB
Data Transfer OUT From Amazon S3 To InternetPrice
Up to 1 GB / Month$0.00 per GB
Next 9.999 TB / Month$0.09 per GB
Next 40 TB / Month$0.085 per GB
Next 100 TB / Month$0.07 per GB
Greater than 150 TB / Month$0.05 per GB
Data Transfer OUT From Amazon S3 to InternetPrice
CloudFront$0.00 per GB
US East (N. Virginia)$0.01 per GB
Asia Pacific (Singapore)$0.02 per GB
Asia Pacific (Sydney)$0.02 per GB
US West (Los Angeles)$0.02 per GB
Canada (Central)$0.02 per GB
Europe (London)$0.02 per GB
Asia Pacific (Tokyo)$0.02 per GB
Europe (Paris)$0.02 per GB
AWS GovCloud (US-East)$0.02 per GB
Europe (Frankfurt)$0.02 per GB
Europe (Ireland)$0.02 per GB
Asia Pacific (Osaka-Local)$0.02 per GB
Asia Pacific (Seoul)$0.02 per GB
Asia Pacific (Mumbai)$0.02 per GB
South America (Sao Paulo)$0.02 per GB
Middle East (Bahrain)$0.02 per GB
AWS GovCloud (US-West)$0.02 per GB
Asia Pacific (Hong Kong)$0.02 per GB
Europe (Stockholm)$0.02 per GB
US West (Northern California)$0.02 per GB
US West (Oregon)$0.02 per GB

Both the storage and bandwidth size include every single file overhead.The prices do not include applicable taxes and duties, such as the VAT and other applicable sales tax. In regard to customers having a Japanese billing address, the usage of their AWS will be subject to the Japanese Consumption Tax.

Amazon S3 storage usage: gets calculated using binary gigabytes [1GB = 230 bytes]. Note that the mentioned unit of measurement is known as well to be a gibibyte, as a definition from the International Electrotechnical Commission. Just as such, 1TB = 240 bytes [i.e. 1024 GBs].

Rate tiers: They take into consideration the aggregate usage for Data Transfer Out to the Internet over all the available AWS services.

Data Transfer Out: It might differ from the data received by a specific app if the connection was prematurely terminated by your own hands, as an example, in case you choose to perform a request for a 10GB object and then just terminate the connection upon receiving your first 2GB of this data. S3 will try to prevent the streaming of data, but it won’t occur in an instantaneous manner. In the mentioned example, Data Transfer Out might reach 3GB (1GB extra than the 2GB you initially got). Because of this, you’re going to get eventually billed for the total of the 3GB of Data Transfer Out.

 

Transfer acceleration

S3 Data Transfer Pricing - transfer acceleration

S3 Data Transfer Pricing – transfer acceleration

Pricing will be based upon the edge location which is utilized for the sake of accelerating your transfer. S3 Transfer Acceleration pricing gets added as well to the Data Transfer pricing.

 

Data Transfer IN to Amazon S3 from the Internet: Cost
Accelerated by AWS Edge Locations in the United States, Europe, and Japan$0.04 per GB
Accelerated by all other AWS Edge Locations$0.08 per GB
Data Transfer OUT from Amazon S3 to the Internet:
Accelerated by any AWS Edge Location		$0.04 per GB
Accelerated by any AWS Edge Location
Data Transfer between Amazon S3 and another AWS region:
Accelerated by any AWS Edge Location
$0.04 per GB

 

You are capable of checking your performance  through the usage of the available Amazon S3 Transfer Acceleration speed comparison tool.Every single time S3 Transfer Acceleration is used for uploading an object, the service is going to be checked for seeing if its possibly capable of being faster than a normal S3 transfer. In case it was determined that it’s not probably going to be faster than a normal S3 transfer of that exact object to the exact destination Region, this utilization of S3 Transfer Acceleration will not get charged for that transfer, and it might be able to actually bypass the S3 Transfer Acceleration system for this specific upload.
The prices do not include applicable taxes and duties, such as the VAT and other applicable sales tax. In regard to customers having a Japanese billing address, the usage of their AWS will be subject to the Japanese Consumption Tax.

AWS Free Usage Tier

S3 Data Transfer Pricing - free tier

S3 Data Transfer Pricing – free tier

When coming to utilize the AWS Free Usage Tier, you’re going to be able to start using Amazon S3 for free. As soon as you tend to sign-up, being a new AWS customer allows you to receive the following:

– 5GB of Amazon S3 storage in the S3 Standard storage class

– 20,000 GET Requests

– 2,000 PUT, POST, LIST, or COPY Requests

– 15GB of Data Transfer Out every single month for a whole year

Usage of your free tier will get calculated every single month across each Region excluding the GovCloud Region and will be directly added straight to your bill; unused monthly usage which remains unused shall not be rolled over, and certain restrictions will apply.

ec2 to s3 data transfer cost

s3 pricing calculator

EC2 to S3 Data Transfer

EC2 to S3 Data Transfer Costs

Posted on | by Ozzy Brightrose | Leave a Comment on EC2 to S3 Data Transfer Costs

EC2 to S3 Data Transfer Costs

There will be no data transfer cost whatsoever when you decide to transfer data out of EC2 to the following service, while remaining within the same region:

– Amazon S3

Without taking into consideration which region or availability zone used, there won’t be any data transfer cost for transferring data to CloudFront or upon the usage of a private IP address.

EC2 to S3 Data Transfer Costs - transfer speed

EC2 to S3 Data Transfer Costs – transfer speed

Improve transfer speeds of copying data between S3 bucket and EC2 instance

 

How to enhance transfer speeds for transferring data from EC2 instance to an S3 bucket?

Transfer speeds for moving, syncing or copying data from EC2 to S3 are dependent upon multiple factors. The below listed methods are the most idle practices for the sake of enhancing the transfer speed while copying, syncing or moving your data between EC2 instance and S3 bucket:

– Enhanced networking on EC2 instance

– Parallel workloads for data transfers

– Upload configurations customization on the CLI

– VPC endpoint for S3

– S3 Transfer Acceleration utilization between geographically distant Regions

– EC2 instance type Upgrading

– Referring to chunked transfers

 

How can each way help?

EC2 to S3 Data Transfer Costs - enhanced networking

EC2 to S3 Data Transfer Costs – enhanced networking

  1. Enhanced networking on EC2 instance

Enhanced networking is capable of providing the following enhancements:

– Higher bandwidth

– Higher PPS performance

– Less inter-instance latencies

It may be enabled for no extra charge.

In the case that your EC2 instance’s PPS rate reaching its highest ceiling, it would have reached the top thresholds for its virtual network interface driver. In case this occurs, try to go ahead and enable enhanced networking.

  1. Parallel workloads for data transfers

For the sake of enhancing your overall time for completing your transfers, try to split the transfer into various mutually exclusive operations. For instance, in the case that you end to utilize the CLI, you are going to be capable of running a number of instances of s3 copy, s3 move, or s3 synchronize all together at similar times.

Important: To get an idle performance, make sure that you use the latest version of your CLI.

  1. Upload configurations customization on the CLI

Customize the below lister CLI configurations for your S3 in aims of speeding up your data transfers:

– multipart_chunksize: Sets size of every part uploaded by the CLI in a multipart upload for individual files. It provides you with the ability to start breaking down larger files into little parts in the sake of getting a quicker upload speed.

– max_concurrent_requests: CLI will support multithreading (By default). It’s possible for you to change this value for the sake of increasing your number of requests sent to S3 at a specifically chosen time. Default value= 10. Upon increasing this value of 10, it’s possible that you’ll be getting a stagnant response. Yet, upon combining higher max_concurrent_requests values while maintaining parallel workloads, you will be able to reach better transfer speeds as an overall result.

  1. VPC endpoint for S3

In case the chosen EC2 instance lies in the same Region as that of the chosen S3 bucket, you are going to have to refer to the usage of a VPC endpoint for S3. Such endpoints are capable of aiding in the enhancement of your overall performance and the reduction of your NAT load.

  1. S3 Transfer Acceleration utilization between geographically distant Regions

In case your EC2 instance and your S3 bucket are both geographically located in closer regions to one another, you will get higher data transfer speeds. In case they are in geographical Regions that are far from each other, you will have to enable S3 Transfer Acceleration. This will provide you with a fast and secure transfer for your longer distances through the use of CloudFront’s edge locations.

  1. EC2 instance type Upgrading

In case you have a high EC2 instance CPU utilization, it’s possible it will become a bottleneck to your whole transfer speed. You are capable of upgrading your instance to a new different instance type which can provide you with higher memory along with better network performance. Instance sizes that are larger than other can offer even stronger network performance than that of the smaller instance sizes being same type.

  1. Referring to chunked transfers

In case you choose to transfer some huge files, you may get aided through using either multipart uploads or ranged GETs for the sake of improving your overall transfer performance.

cloudfront data transfer pricing

Crawlers on Glue Console

Crawlers on Glue Console

Posted on | by Ozzy Brightrose | Leave a Comment on Crawlers on Glue Console

This article provides a detailed overview of using Glue crawlers and how to add a new crawler using AWS Console.

Crawlers on Glue Console

A crawler is normally used to do the following:

  • Accesses your data store.
  • Extracting metadata.
  • Creating table definitions in the Glue Data Catalog.

The Crawlers pane: Tends to list all the created crawlers. This list will display status and metrics associated with the last time you ran the crawler.

Adding a crawler through the console
  • Login to AWS Management Console and navigate to Glue console. Select Crawlers from your navigation pane.

    Crawlers on Glue Console - select

    Crawlers on Glue Console – select

  • Click on Add crawler, then go ahead with following the instructions found in the Add crawler.
Crawlers on Glue Console - add crawler

Crawlers on Glue Console – add crawler

Additional Information
  • For checking out a step-by-step guide with the process of adding a new crawler, click on the Add crawler option listed under Tutorials from your navigation pane.
  • It’s also possible to refer Add crawler wizard for creating and modifying an IAM role which can attach a policy including permissions for Amazon S3 data stores.
  • Otherwise, you’re capable of tagging the crawler using a Tag key and an optional Tag value.
  • Upon getting it created, your tag keys are going to be read-only.
  • You may refer to utilizing tags on specific resources for the sake of aiding you with organizing and identifying them.
  • You are optionally capable of adding a security configuration to a chosen crawler for specifying your at-rest encryption options.
  • Upon running a crawler, you will need to give permission to the provided IAM role so that it can gain access to the data store which is crawled.
  • For an S3 data store, you’re capable of utilizing the Glue console for the sake of creating a policy or adding a policy which looks close to the one shown below:
{
   “Version”: “2012-10-17”,
    “Statement”: [
        {
          “Effect”: “Allow”,
          “Action”: [
              “s3:GetObject”,
              “s3:PutObject”
          ],
          “Resource”: [
              “arn:aws:s3:::bucket/object*”
          ]
        }
    ]
}
  • In case your crawler is capable of reading KMS encrypted S3 data, this will mean that consequently, the IAM role will need to get decrypt permission on this KMS key.
  • Using the Glue console for creating a policy or adding one which is similarly close to the below shown:
  • The following policy can be referred for a DynamoDB data store:
{
  “Version”: “2012-10-17”,
  “Statement”: [
    {
      “Effect”: “Allow”,
      “Action”: [
        “dynamodb:DescribeTable”,
        “dynamodb:Scan”
      ],
      “Resource”: [
        “arn:aws:dynamodb:region:account-id:table/table-name*”
      ]
    }
  ]
}
  • S3 data stores have an exclude pattern which is relative to the include path.
  • As you are crawling a JDBC data store, you will be required to set a connection.
  • The excluded path is going to be relative to the include path.

Ex: Excluding a table in a JDBC data store, you simply have to write down the name of this table in the exclude path.

Upon crawling DynamoDB tables, you will be able to select 1 table name out of the listed DynamoDB tables found in the account.

How to View Crawler Results?
  • To view results of the specific crawler, fetch the name of this crawler from the list and then select the Logs link.
Crawlers on Glue Console - logs

Crawlers on Glue Console – logs

  • By clicking on this link, you will be redirected to AWS CloudWatch Logs service Webpage.
  • There details shown regarding tables were from Glue Data Catalog and the errors which may have encountered.
  • It’s possible to go ahead with managing the log retention period using AWS CloudWatch console.
  • Your default log retention is going to be the following: Never Expire.
  • For checking out details of a specific crawler, click on the name of this crawler in the list.

Crawler details: Information defined upon the creation of this crawler using the Add crawler wizard.

Upon the completion of a crawler run, select Tables from the navigation pane for the sake of viewing the tables which your crawler created in the database specified by you.

Crawlers on Glue Console - aws glue

Crawlers on Glue Console – aws glue

  • Below are a couple of significant properties and metrics related to your last run of a chosen crawler:
  • Name: For newly created crawler, needs to get a unique and special name.
  • Schedule: Your crawler may either be run on demand or on a frequency using a schedule.
  • Status: The state of a crawler which may either be schedule paused, starting, scheduled, ready or stopping. When a crawler is running is gets to progress from a starting status to a stopping one. A schedule which is attached to the crawler may be resumed or paused.
  • Logs: Available logs’ links out of the last run.
  • Last runtime: Period of time a crawler needed for running when it ran the last time.
  • Median runtime: Median period of time a crawler needed for running from when it got created.
  • Tables updated: Number of updated tables in the Glue Data Catalog through the most recent run of a crawler.
  • Tables added: Number of added tables into the Glue Data Catalog through the most recent run of a crawler.
Here are few awesome resources on AWS Services:
AWS S3 Bucket Details
AWS Glue Tags
AWS S3 File Explorer
  • CloudySave is an all-round one stop-shop for your organization & teams to reduce your AWS Cloud Costs by more than 55%.
  • Cloudysave’s goal is to provide clear visibility about the spending and usage patterns to your Engineers and Ops teams.
  • Have a quick look at CloudySave’s Cost Caluculator to estimate real-time AWS costs.
  • Sign up Now and uncover instant savings opportunities.

 

Posted in S3