質問55、 組織では、BigQuery に非常に個人的なデータを保存しており、厳格なデータプライバシー規制を遵守する必要があります。従業員が退職する際には、機密データの値が読み取り不可能になるようにする必要があります。どうすればよいでしょうか？
Comprehensive and Detailed Explanation:
The core requirement is to make specific data values permanently unreadable, a concept often referred to as cryptographic erasure or “crypto-shredding.” Option C is the correct answer because it achieves this at a granular, field-level. BigQuery’s AEAD (Authenticated Encryption with Associated Data) functions allow you to encrypt individual field values within your tables using a keyset from Cloud KMS. You can encrypt the sensitive data, store the encrypted value in BigQuery, and when an employee leaves, you simply destroy the specific KMS key version used for their data. Once the key is destroyed, the encrypted data is permanently unrecoverable, thus rendering it unreadable.
Option A and B are incorrect because data masking and column-level access controls are about controlling access to the data, not about making the underlying data itself unreadable. If an administrator with sufficient privileges were to query the table, the data would still be there in its original form. Revoking permissions prevents a specific user from seeing the data, but it doesn’t erase it.
Option D is incorrect because Customer-Managed Encryption Keys (CMEK) operate at the entire table level.
Deleting a CMEK key would make the entire table unreadable, which is far too broad and disruptive. The goal is to target specific sensitive values, not destroy whole tables.
Reference (Google Cloud Documentation Concepts):
This solution leverages field-level encryption within BigQuery. The official documentation for “Protecting data with Cloud KMS keys” states that BigQuery’s AEAD functions (AEAD.ENCRYPT and AEAD.
DECRYPT) are the intended mechanism for this purpose. This approach provides fine-grained control, allowing you to manage the lifecycle of encrypted data by managing the lifecycle of the encryption keys in Cloud Key Management Service (Cloud KMS). The act of destroying a key version is the mechanism for crypto-shredding.
質問56、 あなたは服の推奨をするためのモデルを構築しています。ユーザーのファッションの好みは時間の経過とともに変化する可能性が高いことがわかっているため、データパイプラインを構築して、新しいデータが利用可能になったときにモデルにストリーミングします。
このデータをどのように使用してモデルをトレーニングする必要がありますか？
https://cloud.google.com/automl-tables/docs/prepare
Topic 1, Flowlogistic Case Study
Company Overview
Flowlogistic is a leading logistics and supply chain provider. They help businesses throughout the world manage their resources and transport them to their final destination. The company has grown rapidly, expanding their offerings to include rail, truck, aircraft, and oceanic shipping.
Company Background
The company started as a regional trucking company, and then expanded into other logistics market. Because they have not updated their infrastructure, managing and tracking orders and shipments has become a bottleneck. To improve operations, Flowlogistic developed proprietary technology for tracking shipments in real time at the parcel level. However, they are unable to deploy it because their technology stack, based on Apache Kafka, cannot support the processing volume. In addition, Flowlogistic wants to further analyze their orders and shipments to determine how best to deploy their resources.
Solution Concept
Flowlogistic wants to implement two concepts using the cloud:
Use their proprietary technology in a real-time inventory-tracking system that indicates the location of their loads
Perform analytics on all their orders and shipment logs, which contain both structured and unstructured data, to determine how best to deploy resources, which markets to expand info. They also want to use predictive analytics to learn earlier when a shipment will be delayed.
Existing Technical Environment
Flowlogistic architecture resides in a single data center:
Databases
8 physical servers in 2 clusters
SQL Server – user data, inventory, static data
3 physical servers
Cassandra – metadata, tracking messages
10 Kafka servers – tracking message aggregation and batch insert
Application servers – customer front end, middleware for order/customs
60 virtual machines across 20 physical servers
Tomcat – Java services
Nginx – static content
Batch servers
Storage appliances
iSCSI for virtual machine (VM) hosts
Fibre Channel storage area network (FC SAN) – SQL server storage
Network-attached storage (NAS) image storage, logs, backups
Apache Hadoop /Spark servers
Core Data Lake
Data analysis workloads
20 miscellaneous servers
Jenkins, monitoring, bastion hosts,
Business Requirements
Build a reliable and reproducible environment with scaled panty of production.
Aggregate data in a centralized Data Lake for analysis
Use historical data to perform predictive analytics on future shipments
Accurately track every shipment worldwide using proprietary technology
Improve business agility and speed of innovation through rapid provisioning of new resources
Analyze and optimize architecture for performance in the cloud
Migrate fully to the cloud if all other requirements are met
Technical Requirements
Handle both streaming and batch data
Migrate existing Hadoop workloads
Ensure architecture is scalable and elastic to meet the changing demands of the company.
Use managed services whenever possible
Encrypt data flight and at rest
Connect a VPN between the production data center and cloud environment
SEO Statement
We have grown so quickly that our inability to upgrade our infrastructure is really hampering further growth and efficiency. We are efficient at moving shipments around the world, but we are inefficient at moving data around.
We need to organize our information so we can more easily understand where our customers are and what they are shipping.
CTO Statement
IT has never been a priority for us, so as our data has grown, we have not invested enough in our technology. I have a good staff to manage IT, but they are so busy managing our infrastructure that I cannot get them to do the things that really matter, such as organizing our data, building the analytics, and figuring out how to implement the CFO’ s tracking technology.
CFO Statement
Part of our competitive advantage is that we penalize ourselves for late shipments and deliveries. Knowing where out shipments are at all times has a direct correlation to our bottom line and profitability. Additionally, I don’t want to commit capital to building out a server environment.
質問58、 Dataflow パイプラインを使用して注文データをストリーミングし、集計結果を Memorystore に書き込みます。Memorystore for Redis インスタンスを Basic Tier でプロビジョニングしました。4 GB の容量。40 台のクライアントが読み取り専用アクセスに使用します。読み取り専用クライアントの数が数百にまで大幅に増加すると予想されており、その需要に対応できる必要があります。読み取りおよび書き込みアクセスの可用性に影響を与えず、加えた変更をすぐに展開できるようにしたいと考えています。あなたは何をするべきか？
The Basic Tier of Memorystore for Redis provides a standalone Redis instance that is not replicated and does not support read replicas. This means that it cannot scale horizontally to handle more read requests, and it does not provide high availability or automatic failover. If the number of read-only clients increases significantly, the Basic Tier instance may not be able to handle the demand and may impact the read and write access availability. Therefore, option A is not a good solution, as it would require creating multiple Basic Tier instances and modifying the Dataflow pipeline and the clients to distribute the load among them. This would increase the complexity and the management overhead of the solution.
The Standard Tier of Memorystore for Redis provides a highly available Redis instance that supports replication and read replicas. Replication ensures that the data is backed up in another zone and can fail over automatically in case of a primary node failure. Read replicas allow scaling the read throughput by adding up to five replicas to an instance and using them for read-only queries. The Standard Tier also supports in-transit encryption and maintenance windows. Therefore, option D is the best solution, as it would create a new Standard Tier instance with a higher capacity (5 GB) and multiple read replicas to handle the increased demand. The old instance can be deleted after migrating the data to the new instance.
Option B is not a good solution, as it would create a new Standard Tier instance with the same capacity (4 GB) and no read replicas. This would not improve the read throughput or the availability of the solution. Option C is not a good solution, as it would create a new Memorystore for Memcached instance, which is a different service that uses a different protocol and data model than Redis. This would require changing the code of the Dataflow pipeline and the clients to use the Memcached protocol and data structures, which would take more time and effort than migrating to a new Redis instance. References: Redis tier capabilities | Memorystore for Redis | Google Cloud, Pricing | Memorystore for Redis | Google Cloud, What is Memorystore? | Google Cloud Blog, Working with GCP Memorystore – Simple Talk – Redgate Software
