Splunk Enterprise Software

What is Splunk?

Splunk Enterprise Software (“Splunk”) is a powerful tool for searching and exploring data.

Splunk is often used by system administrators, network administrators, and security gurus, but its use is not restricted to these audiences. There is a great deal of business value hidden away in corporate data that Splunk can liberate.

Who uses Splunk?

Splunk is a powerful platform for analyzing machine data, data that machines emit in great volumes but which is seldom used effectively. Machine data is already important in the world of technology and is becoming increasingly important in the world of business.

It’s used by:

  • Security offices
  • Marketing departments
  • System administrators
  • Network administrators
  • Application development teams
  • Application support teams

splunk_usage

How does it work?

  • Splunk begins with indexing, which means gathering all the data from diverse locations and combining it into centralized indexes.
  • Using the indexes, Splunk can quickly search the logs from all servers and hone in on when the problem occurred.
  • Splunk can then drill down into the time period when the problem first occurred to determine its root cause. Alerts can then be created to head the issue off in the future.

Splunk provides one repository, data indexing, search & visualization for your data.

What are the potential Splunk data sources?

During indexing, Splunk can read machine data from any number of sources. The most common input sources are:

  • files: Splunk can monitor specific files or directories. If data is added to a file or a new file is added to a monitored directory, Splunk reads that data.
  • the network: Splunk can listen on TCP or UDP ports, reading any data sent.
  • scripted inputs: Splunk can read the machine data output by programs

or scripts, such as a Unix® command or a custom script that monitors sensors.
Technically speaking, retrieved events from your indexes are called “events.” If those events are transformed or summarized so that there is no longer a one-tone mapping with events on disk, they are properly called “results.”

Announcements at the Splunk 2015 Conference in a slide

splunk_news

SPL (Search Processing Language)

The Search Processing Language encompasses all the search commands and their functions, arguments and clauses. Search commands tell Splunk Enterprise what to do to the events you retrieved from the indexes. For example, you need to use a command to filter unwanted information, extract more information, evaluate new fields, calculate statistics, reorder your results, or create a chart.

Some search commands have functions and arguments associated with them. Use these functions and their arguments to specify how the commands act on your results and/or which fields they act upon. For example, use functions to format the data in a chart, describe what kind of statistics to calculate, and specify what fields to evaluate. Some commands also use clauses to specify how to group your search results.

There are four broad categorizations for all the search commands: distributable streaming, stateful streaming, transforming, generating.

splunk_spl

Where can I find more information?

http://docs.splunk.com
http://docs.splunk.com/Documentation/Splunk
http://splunkbase.com
http://docs.splunk.com/images/a/a3/Splunk_4.x_cheatsheet.pdf
http://answers.splunk.com/
Reference Guide http://www.splunk.com/web_assets/pdfs/secure/Splunk_Quick_Reference_Guide.pdf
Educational Videos http://www.splunk.com/view/education-videos/SP-CAAAGB6
Splunk Blogs http://blogs.splunk.com/
Splunk Wiki http://wiki.splunk.com/Special:SplunkSearch/wiki?q=your-query
Certification and education http://www.splunk.com/view/education/SP-CAAAAH9

Print Friendly

Elastic Search

Elastic Search is a document oriented search engine based on JSON & Apache Lucene (java library). If I have to explain in plain language Elastic Search, I would tell you it’s a search engine, algorithm, technology to search big amounts of data(documents) in a very optimized way. Of course the easiest is to tell you it’s a Solr like technology with a bit more powerful search and aggregation capabilities. It can be invoked from a web application or by a process. You can define search parameters or indexing.

 In Elastic Search you can do structured search (filters), full-text search (query), and analytics (aggregations).  

When using analytics (aggregations) you need to add more complex aggregations, percolators and may be scripts.

It’s schemaless (schemas are created dynamically). You dont need to define in advance the structure of the data you are feeding for indexing.

There are free plugins like Kibana you can use to monitor your Elastic Search servers and performance.

Kibana

Elastic Search uses log4j for logging application errors and I would like to see that configured to log to a database as well as the local file.

How do we run queries against Elastic Search? There are multiple ways to do that. A favorite option is Postman. https://chrome.google.com/webstore/detail/postman-rest-client/fdmmgilgnpjigdojojpjoooidkmcomcm?hl=en or Kibana Sense.

Application Support Elastic Search role skills/knowledge needed:

– Elastic Search DSL (Domain specific language) is the query language in Elastic Search (for debugging)
– JSON
– Elastic Search understanding
– Kibana understanding, for monitoring/reporting and configuring monitoring/reporting
– Knowledge of the implementation
– AWS/ Apache Lucene Java library/ Linux / JVM
– Knowledge of the server/application setup

laptop

Print Friendly

Microsoft System Center

Microsoft System Center is a suite of tools to help you manage physical and virtual servers, client computers and devices.

There are different components of System Center. In 2012 it became a single product. You can use any of the products with one license.

  1. VM Manager
  2. Service Manager
  3. Orchestrator (Azure Orchestration)
  4. Configuration Manager
  5. Data Protection Manager
  6. App Controller
  7. Endpoint Protection
  8. Operations Manager

Microsoft System Center is changing and evolving.

  • Changing potentially to a service on the Cloud
  • 1 in 5 machines in Azure currently is on Linux and Microsoft is trying to accommodate them

What Does Microsoft System Center Do?

It’s an evolution (started as managing desktops, became a configuration manager). In 2012 it became a standalone product. It’s cross platform – it works with Azure, AWS- it’s  agnostic.

  • Patching on cross platforms
  • Deployments
  • Infrastructure monitoring
  • Application performance monitoring
  • Dynamic application discovery (with Blue stripe)- it sniffs ports, with a light agent, the physical mapping of the application
  • SQL backups
  • Azure backups
  • Security and audit
  • DR
  • Automation
  • Updates client computers and devices across physical, virtual, distributed and mobile environments

What is Operations Management Suite (OMS)?

It sits on top of System Center. It can be on the Cloud or on the premise. It requires an additional license. OMS offers:

  • Log Analytics
  • You can do custom dashboards
  • Ties easily AWS & Azure
  • Nice web platform/view (HTML5)

It has 260+ business intelligence packs (plugins).

OMS

Print Friendly

AWS Abbreviations

SQS- Simple Queuing Service

SNS- Simple Notification Service

WAM- Workspace Application Manager

VPC – Virtual Private Cloud

DC (DX)- Direct Connect

EFS – Elastic Filesystem

VPE – Virtual Private Endpoint (?)

NAT – Network address translation

EMR- Amazon Elastic MapReduce (EMR) is a web service that uses Hadoop, an open-source framework, to quickly & cost-effectively process vast amounts of data.

IOPS- Input/Output Operations per Second

NFS – Network File System

AML – Amazon Machine Learning

NAS – Network Attached Storage

S3 – Simple Storage Service

EC2- Amazon Elastic Compute Cloud

RDS – Amazon Relational Database Service

ELB – Elastic Load Balancing

Aurora – Amazon’s MySQL-compatible relational database management system (RDBMS)

Print Friendly

Right Scale – Manage Your Cloud

Right Scale is a tool to help us with managing a multi-cloud environment. It’s a single interface for multi-cloud support (example AWS, Azure, Google Cloud, as well as VM).

Right Scale is SaaS (software as a service).

The features:
– Cloud and configuration management
– Credential management
– One place to manage user access to the multi-cloud environment
– Managing and automating server patching & maintenance
– Server deployments automation
– Monitoring & alerting (including backup alerts)
– Cloud cost management (cloud analytics)
– With the Collectd plugin you can collect server statistics (in numbers)
– There is the  Multi Cloud Marketplace which gives you access to server templates and scripts (ex. Chef recipes, right script and so on)
– RightScale APIs (everything that’s in the interface, can be done through API)
– It has Chef integration. It has New Relic integration.

The main focus of Right Scale:
* Automation (in and across Clouds through Self-Service and CAT files)
* Governance (Offer self-service IT access while sleeping at night. Design and enforce policies based on budgets, configurations, and user access. Control resource placement and capacity.)
* Economics (Through Cloud Analytics and Self-Service Scheduling we are able to control waste)
* Ecosystem (Multi-cloud & hybrid environment)

Note: Hybrid is the combination of private Cloud (VMware, Open Stack & Cloud Stack environment) and public Cloud (ex. AWS, Azure, Google…).

There are 3 parts to Right Scale Cloud Portfolio Management:
Self-Service, Cloud Management, and Cloud Analytics

RightScale1

Cloud Analytics – shows the cost of the cloud (User oriented view)

  • Budget Alerts
  • Spending Reports
  • Estimation Tool (Scenario Builder)

RightScale2

Want to read more? www.rightscale.com

Print Friendly