Read CSV File Using Python Panda and Store To DB

Nice blog

deepakrdash's avatarRuby on Rails and Python Django

Pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.

Python with Pandas is used in a wide range of fields including academic and commercial domains including finance, economics, Statistics, analytics, etc. In this tutorial, we will learn the various features of Python Pandas and how to use them in practice.

Advantages

  • Fast and efficient for manipulating and analyzing data.
  • Data from different file objects can be loaded.
  • Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data
  • Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects
  • Data set merging and joining.
  • Flexible reshaping and pivoting of data sets
  • Provides time-series functionality.
  • Powerful group by functionality for performing split-apply-combine operations on data sets.

In this blog I will be showing you how to

View original post 249 more words

Read CSV File Using Python Panda and Store To DB

Pandas is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.

Python with Pandas is used in a wide range of fields including academic and commercial domains including finance, economics, Statistics, analytics, etc. In this tutorial, we will learn the various features of Python Pandas and how to use them in practice.

Advantages

  • Fast and efficient for manipulating and analyzing data.
  • Data from different file objects can be loaded.
  • Easy handling of missing data (represented as NaN) in floating point as well as non-floating point data
  • Size mutability: columns can be inserted and deleted from DataFrame and higher dimensional objects
  • Data set merging and joining.
  • Flexible reshaping and pivoting of data sets
  • Provides time-series functionality.
  • Powerful group by functionality for performing split-apply-combine operations on data sets.

In this blog I will be showing you how to retrieve data from a csv file using panda and store it’s values in a database using Django REST API.

Lets get started

To get started run the following command, it will install django, django-rest and pandas which we will be using to read our csv file

pip install django djangorestframework pandas

In your Djnago models.py file create a new table Product like below

class Product(models.Model):
    product_name = models.CharField(max_length=100)
    product_quantity= models.CharField(max_length=200)
    product_price = models.IntegerField() 
    
    def __str__(self):
        return self.product_name 

The above code will create a Product table and store the product information from csv file

Next create a serializers.py in the same application and add the following

from rest_framework import serializers
# remember to import the File modelclass FileUploadSerializer(serializers.Serializer):
    file = serializers.FileField()
class SaveFileSerializer(serializers.Serializer):
    
    class Meta:
        model = Product
        fields = "__all__"

FileUploadSerializer is what will allow user’s upload the csv file. Next add the following in views.py file of the same application

from django.shortcuts import render
from rest_framework import generics
import io, csv, pandas as pd
from rest_framework.response import Response
from api import models

class FileUploader(generics.CreateAPIView):
    serializer_class = FileUploadSerializer
    def post(self, request, *args, **kwargs):
        serializer = self.get_serializer(data=request.data)
        serializer.is_valid(raise_exception=True)
        file = serializer.validated_data['file']        
        reader = pd.read_csv(file, keep_default_na=False,na_values=" NaN")
        for _, row in reader.iterrows():
            product = Product(
                       product_name= row["Product Name"],
                       product_quantity= row['Quantity'],
                       product_price= row["Amount"]
                       )
            product.save() 
       return Response({"status": "success"},status.HTTP_201_CREATED)

Finally we are all set, you can go ahead and start your django server, and uplaod the csv file.

I hope you found this short blog helpful. Thanks for reading to the end.

Django Workflow and Architecture

What is Django?

Django is a free and open source web application framework written in Python. A framework is nothing more than a collection of modules that make development easier. They are grouped together, and allow you to create applications or websites from an existing source, instead of from scratch.

When you’re building a website, you always need a similar set of components: a way to handle user authentication (signing up, signing in, signing out), a management panel for your website, forms, a way to upload files, etc.

Frameworks exist to save you from having to reinvent the wheel and to help alleviate some of the overhead when you’re building a new site and Django framework is one of them.

The official project site describes Django as “a high-level Python Web framework that encourages rapid development and clean, pragmatic design. Built by experienced developers, it takes care of much of the hassle of Web development, so you can focus on writing your app without needing to reinvent the wheel. It’s free and open source.”

Django offers a big collection of modules which you can use in your own projects. Primarily, frameworks exist to save developers a lot of wasted time and headaches and Django is no different.

Django Architecture

Django follows the MVT framework for architecture.

  • M stands for Model
  • V stands for View
  • T stands for Template

MVT is generally very similar to that of MVC which is a Model, View, and Controller. The difference between MVC and MVT here is the Django itself does the work done by the controller part in the MVC architecture. Django does this work of controller by using templates. Precisely, the template file is a mixture of HTML part and Django Template Language also known as DTL.

Benefits of Django Architecture

The Django Framework is based on this architecture and it actually communicates between all these three components without needing to write complex code. That’s why Django is gaining popularity.

This architecture in Django has various advantages like:

  1. Rapid Development: Django architecture that separates in different components makes it easy for multiple developers to work on different aspects of the same application simultaneously. That is also one of the features of Django.
  2. Loosely Coupled: Django has different components which require each other at certain parts of the application, at every instant, that increases the security of the overall website. As the model file will now only save on our server rather than saving on the webpage.
  3. Ease of Modification: This is an important aspect of development as there are different components in Django architecture. If there is a change in different components, we don’t have to change it in other components. This is actually one of the special features of Django, as here it provides us with much more adaptability of our website than other frameworks.
  4. Security: While building high-end web applications, security becomes a very crucial aspect to ponder on. Django understands this concern and provides its best defender to save your efforts.Using click-jacking, cross-site scripting, SQL injections Django builds a strong wall for the application’s safety. User authentication is also an important feature to safely manage user accounts and passwords that Django provides.
  5. Scalable: Scalability can be understood as an ability of an application to work well when the size or volume of the platform increases.Django works effortlessly in this issue. Websites made with Django have the capacity to handle multiple users at a single time. Some popular websites using Django include Spotify, Netflix, YouTube, Mozilla, Quora, etc

Creating A New Project In Django

To start a new Django project, run the command below:

django-admin startproject myproject

After we run the command above, it will generate the base folder structure for a Django project.

Right now, our myproject directory looks like this:

myproject/                  <– higher level folder |– myproject/            
 |    |– myproject/ <– django project folder
 |    |    |– __init__.py
 |    |    |– settings.py
 |    |    |– urls.py
 |    |    |– wsgi.py
 |    +– manage.py
  • manage.py: a shortcut to use the django-admin command-line utility. It’s used to run management commands related to our project. We will use it to run the development server, run tests, create migrations and much more.
  • __init__.py: this empty file tells Python that this folder is a Python package.
  • settings.py: this file contains all the project’s configuration. We will refer to this file all the time!
  • urls.py: this file is responsible for mapping the routes and paths in our project. For example, if you want to show something in the URL /about/, you have to map it here first.
  • wsgi.py: this file is a simple gateway interface used for deployment.

Bootstrap Theme Integration From Scratch In Ruby On Rails

boots_rails

This blog gives you the fundamental idea of integrating different bootstrap themes for your new app.

Below are the steps to integrate bootstrap themes

Step:1

Create a new rails application. Go to the Gemfile and the following gems

gem 'twitter-bootswatch-rails'
gem 'twitter-bootswatch-rails-helpers'

Run bundle to install the above gems and it’s dependencies.

Step:2

Go to the console and run the following commands to install and import different bootstrap themes of your choice. You can get a wide variety of free theme names from https://bootswatch.com  . Click on the “Themes” dropdown to get the theme names


rails g bootswatch:install cerulean 
rails g bootswatch:import cerulean 
rails g bootswatch:layout cerulean

In above command cerulean is an available theme name. You can use any such as ‘cosmo’, ‘lumen’,’sandstone’ etc from the site…

Step:3

Go to your application_controller.rb file and add the following line

layout 'cerulean'

Now your application will use the cerulean theme layout instead of application.html.erb.

Step:4

Go to config\initializers and open the file assets.rb file. Add the css and js files for the installed theme.

Rails.application.config.assets.precompile += %w( cerulean.css cerulean.js)

Restart the server and refresh your browser to view the installed theme UI.

Hope it will help you.

Change Position By Drag/Drop Using Jquery In Ruby On Rails

drag  This blog will show how to easily change the position of lists by drag and drop method using jquery plugin.For this we have to use the TableDnD jquery plugin,which you can download from this URL “https://github.com/isocra/TableDnD/”.

Here are the steps for integrating to change the position by drag and drop using jquery.

Step: 1

          Download the jquery plugin from above URL.You have to use “jquery.js”,”jquery.tablednd.0.6.min.js” and “jquery.tablednd.js” files.Add those js files in your application.

Step: 2

     Your html view file should be like below format

    <table id=”articles” width=”100%”>
     <tr>
         <th>
      Name
    </th>
    <th>
      Created
    </th>
    <th>
      Action
    </th>
  </tr>
  <tbody class=”appsHeader1″ style=”overflow:hidden;”>
    <% @articles.each do |a| %>
     <tr class=”<%= cycle(“even”, “odd”) %>” id=”art-<%= a.id %>” onMouseOut=”this.bgColor=’#fff’;” onMouseOver=”this.bgColor=’#9CB071′;” style=”font-size:12px; width:875px;”>
        <td>
          xxxxx
        </td>
        <td>
         xxxx
        </td>
        <td>
          xxxxx
        </td>
      </tr>
    <% end %>
  </tbody>
</table>

Step: 3

   Add the following jquey code to update the current position.

   <script charset=”utf-8″ type=”text/javascript”>
  $(document).ready(function() {
  $(‘#articles’).tableDnD({
  onDragClass: “selRow”,
  onDrop: function(table, row) {
  $.ajax({
  type: “POST”,
  url: “#{url_for(:action => ‘sort’)}”,
  processData: false,
  data: $.tableDnD.serialize() + ‘&authenticity_token=’ + encodeURIComponent(‘#{form_authenticity_token if protect_against_forgery?}’),
  success: function(msg) {
  }
  });
  }
  })
  })
</script>

Step: 4

In your controller write the sort method.

 def sort
  Article.all.each do |a|
      if position = params[:articles].index(a.id.to_s)
        a.update_attribute(:position, position + 1) unless a.position ==  position + 1
      end
    end
    render :nothing => true, :status => 200
  end

Also change your routes accordingly.Now you can easily re-order the position in your rails app.

Import CSV File Into The Database In Rails Application.

CSV

 

In this blog i’ll demonstrate how to import CSV file data into your rails application using Ruby’s built-in CSV library.It allow users to import records into the database by uploading a CSV file.

Below are steps to import the CSV records.

Step:1
Add the following line in application.rb file

require 'csv'

Restart the server.

Step:2
Add the block in your view page for users to upload CSV file.

<%= form_tag home_path, multipart: true do %>
  <%= file_field_tag :file %>
  <%= submit_tag "Import CSV" %>
<% end %>

Step:3
Add the block in your controller

def import
   Users.import(params[:file])
  end

Step:4
Go to user model and write the following code block

def self.import(file)
  CSV.foreach(file.path, headers: true) do |row|
    Users.create! row.to_hash
  end
end

But one thing to remember the column header names of your csv file must be match with the column names of the inserting table.when you reload the page and upload the csv,all the data will save in the database.

Google Maps for Rails with Gmaps4 Rails: Tutorial on how to post and filter locations

andrewglass1's avatarAndy Glass

In this blog post, I’ll review how to geocode and map locations using Gmaps4rails, an awesome gem from Benjamin Roth. I’ll also show you how to use javascript to create filters to show/hide markers using data from your model.

Lets get started

Installing Gmaps4rails
In your Gemfile, add the gem

gem 'gmaps4rails'

Bundle and from your commandline, instruct the below to generate the Gmaps4rails coffeescript files.

$ rails generate gmaps4rails:install

Gmaps4rails works by making a rails model ‘gmappable’. The data I’ll be using for this demo is based on a city model. Lets create the migration:

class CreateCities < ActiveRecord::Migration
  def change
    create_table :cities do |t|
      t.string :name
      t.string :state
      t.float :latitude
      t.float :longitude
      t.boolean :gmaps
      t.integer :population
      t.timestamps
    end
  end
end

To make the model ‘gmappable’, on the model:

acts_as_gmappable

Gmaps4rails has an optional geocoding feature which calculates the lat and long needed to plot a location. We’ll…

View original post 1,450 more words

URL Shortening on Rails 3 with Bit.ly

salayhin's avatarSalayhin

For integrating with twitter you need to shorten url. I have used this gem

First add gem to your gemfile


and run

Now add this to your controller

philnash’s gem has support for the bit.ly version 2 and version 3 api. I have use version 3.

Create a new file config\initializers\bitly.rb and write this.

That’s it from installation.

Here is code from controller. This is the example from the gem documentation

Based on this your result is expected. if you want the short url try

That’s it. Happy coding 🙂

View original post

MUST READ – Stay alert while you travel: True story of awareness!

Din's avatarPROPEL STEPS

darkness

A few days before I along with my family were travelling to our hometown for a marriage. It was a long night journey on a booked Cab and the clock ticked around 2 AM. When we arrived at a dark remote area, I have noticed a car probably met with an accident and seemed disfigured on the roadside. Roughly I could see a woman and a child bleeding heavily, but I am sure they were suffering.

“Have you seen that? Please stop the car” I urged the cab driver. But he turned off the music and drove the vehicle faster. I raised my voice “Look there were a kid and women bleeding to death, we should behave humane and this is not right to leave like this, are you going to stop or not?” this time I was a bit harsh on my sound. “Please calm down Sir, I know what…

View original post 703 more words

Google Integration Using Devise and Omniauth In Rails App.

ruby-on-rails-logoww             Add              118003-matte-blue-and-white-square-icon-social-media-logos-google-g-logo

In this blog i’ll show how to integrate Google authentication using devise.To install devise you can refer to my blog here.

Step:1
Add the gems in your gem file

gem ‘devise’
gem 'omniauth'
gem 'omniauth-google-oauth2' 

Run the “bundle install” command to install the gem.

Step:2
You need two more columns to store provider type and userid given from google

rails g migration AddProviderToUsers provider:string uid:string

Runt rake db:migrate to insert the columns in users table.

Step:3
Go the user model “user.rb” and add the following line

devise : omniauthable

Step:4
First of all you need to create an app in google to get “Client key” and “Client Secret key”

https://code.google.com/apis/console/

Create an app and get the Client id and secret key.

Step:5
Now you need to declare the provider name and client id and key.Go to the file config/initializers/devise.rb and the following line

require 'omniauth-google-oauth2'
config.omniauth :google_oauth2, "APP_ID", "APP_SECRET", { access_type: "offline", approval_prompt: "" }

Step:6
Go to your layout file and the following block


<% if user_signed_in? %>
Signed in as <%= current_user.name %>. Not you?
<%= link_to "Sign out", destroy_user_session_path,:method => :delete %>
<% else %>
<%= link_to "Sign up", new_user_registration_path %> or
<%= link_to "Sign in", new_user_session_path %>
<%= link_to "Sign in with Google", user_omniauth_authorize_path(:google_oauth2) %>
<% end %>

Step:7
Before creating the call back method change your route like below


devise_for :users, :controllers => { : omniauth_callbacks => "omniauth_callbacks" }

Step:8
Create a new controller named as “omniauth_callbacks_controller.rb”.Add the following method in it.


class OmniauthCallbacksController < Devise::OmniauthCallbacksController   
def google_oauth2
     
    @user = User.find_for_google_oauth2(request.env["omniauth.auth"], current_user)
 
    if @user.persisted?
      flash[:notice] = I18n.t "devise.omniauth_callbacks.success", :kind => "Google"
      sign_in_and_redirect @user, :event => :authentication
    else
      session["devise.google_data"] = request.env["omniauth.auth"]
      redirect_to new_user_registration_url
    end
  end

Step:9
Add the following block in your user model.


def self.find_for_google_oauth2(access_token, signed_in_resource=nil)
    data = access_token.info
    user = User.where(:provider => access_token.provider, :uid => access_token.uid ).first
    if user
      return user
    else
      registered_user = User.where(:email => access_token.info.email).first
      if registered_user
        return registered_user
      else
        user = User.create(name: data["name"],
          provider:access_token.provider,
          email: data["email"],
          uid: access_token.uid ,
          password: Devise.friendly_token[0,20],
        )
      end
   end
end

Now we are done with google integration with devise.