Intro to Cloud Pub/Sub
Pub/Sub interacts with Core, Functions, and Dataflow. Pub/Sub is a real time messaging service. Decouples sender and receiver with asynchronous communications. Holds data up to seven das. It does not guarantee first in and first out; so not guaranteed to get things in order. Can be the single ingest for all different data points. Can dynamically rate limits, throttle how often or how much pub/sub will push those messages. Pub/Sub end to end reliability guarantee at least one delivery and receipt of individual message. Everything is encrypted as well. Pay for what you need, maintenance is free. Two levels of indirection between the publisher and subscriber. This de-couples the sender’s transmission of the message from the receiver’s receipt of the message. A message is simply data in transit through the system. The message consists of a payload and optional attributes that describe the payload. The message is published to a specific topic. A topic is a feed of messages. The topic stores the message ensuring availability and reliability. The message is transmitted to one or more subscriptions. Pull subscribers using HTTPS requests to google apis. Push subscribers use webhook endpoints that can accept post requests over HTTPS. The subscription registers each delivery. When all of the deliveries are complete, it removes the message from the queue.
Messages are stored before they are delivered. The subscription responds to the delivery by removing the message from the queue.
Some use cases for Google Pub/Sub include balancing workloads in network clusters. Implementing asynchronous workflows, distributing event notifications, refreshing distributed caches, logging in to multiple systems, data streaming from various processes or devices, and reliability improvement.
Pub/Sub is a message oriented middleware to the cloud. Decoupling senders and receivers allow for secure, highly available communication between devices and services. Pub/Sub ingests event streams and delivers them to Cloud Dataflow. Cloud Dataflow processes the data and delivers it to BigQuery for analysis and storage or to Google Cloud Storage.
Subscribers either pull messages from a subscription or are configured as webhooks for push subscriptions. Every subscriber must acknowledge each message within a configurable window of time.
Learn to set up a topic to hold data, subscribe to a topic to access the data, publish and then consume messages with a pull subscriber.
Introduction to Google Cloud IoT Core:
IoT Core is a 100% managed service; no autoscaling, setup redundancy, database partitioning or resource pre-provisioning. Google IoT combines MQTT protocol with the highest level of security (TLS1.2 with certificates), and it is a single GLOBAL endpoint (mqtt.googleapis.com). You can update and control devices using the device manager.
IoT is taking devices that were not traditionally connected to the internet and then adding connectivity to them. IoT Core provides a way of creating a registry where you can define where devices are and then connect them. Device telemetry is forwarded to a Cloud Pub subtopic You can then use Cloud Functions to trigger API calls or you can use Dataflow to extract that data from pub-sub and put it into a data warehouse such as BigTable or BigQuery. This will allow you to analyze and react to that data in real time.
Cloud IoT provisioning is an add-on service for IoT Core that simplifies device creation and monitoring. It allows you to securely add devices to your network by using hardware based security, also known as crypto chip or secure element (SE) from OEM partners. This enables you to provision millions of devices to the right Cloud IoT core without human intervention.
In order for a device to connect, it must first be registered in the device manager. The device manager can be used through the google cloud platform console, gcloud commands, or the rest-style api. Each device registry is created in a specific cloud region, and belongs to a cloud project. A registry is identified in the cloudiot.googleapis.com service by its full name as: projects/{project-id}/locations/{cloud-region}/registries/{registry-id}.
A device reg is config with one or more cloud pub/sub topics to which telemetry events are published for all devices in that reg. A single topic can be used to collect data across all regions. Stackdriver monitoring is automatically enabled for each registry. Cloud Identity and Access Management (IAM) can be used for access control to view, provision or fully manage devices. IoT Core automatically grants the role gloudiot.serviceAgent to the corresponding service account for each project, in order to enable publishing to pub/sub topics.
Protocol Bridges: MQTT is a pub/sub protocol and is often used with embedded devices. HTTP is a connectionless protocol; when using this protocol devices maintain a connection to cloud IoT Core. Cloud IoT core supports HTTP 1.1. only